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Preface 



For the twenty-ninth year, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the national AECT Convention in Dallas, TX. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. Volume #1 is 
available through the Educational Resources Clearinghouse (ERIC) System. 

The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with instruction and training issues are 
contained in volume #2. 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 
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Editor 
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How Teacher Educators’ Beliefs and Technology Uses Relate to Preservice 
Teachers’ Beliefs and Attitudes toward Technology 

Hua Bai 

The State University of New York at Potsdam 

Peggy Ertmer 
Purdue University 

Introduction 

Beliefs about teaching and learning play an important role in transforming classrooms through the use of 
technology (Ertmer, 1999, 2005). Additionally, attitudes toward technology can influence teachers’ technology uses 
(Myers & Halpin, 2002; Yildirim, 2000). In order to prepare preservice teachers to use technology effectively, it is 
necessary to examine their beliefs about learning and teaching and their attitudes toward technology. Teacher 
educators must act as role models for preservice teachers and prepare them to use technology in their future 
professional practices. In this paper we report the results of a study that examined teacher educators’ beliefs and 
technology uses in relation to preservice teachers’ beliefs and technology attitudes. 

Ertmer (1999) described two types of barriers to technology integration: first-order barriers and second- 
order barriers. First-order barriers are extrinsic to teachers and include lack of access to hardware and software, 
time, and necessary support. Second-order barriers are intrinsic to teachers and include teachers’ belief systems 
about teaching and learning, as well as their familiar teaching practices, which can affect meaningful technology 
integration. Since second-order barriers are more ingrained and personal than first-order barriers, they may be harder 
to overcome. Current literature (Becker, 2000; Niederhauser & Stoddart, 2001; Norm, Grabinger & Duffield, 1999) 
indicated that teachers who had more constructivist or student-centered beliefs tended to use technology and engage 
the students in more technology-supported student-centered learning activities, while those who had more traditional 
or teacher-centered beliefs tended to use less technology and had students use technology to reinforce skills. 

Some researchers (Cifuentes, 1997; Daniel, 1996; Friel & Carboni, 2000; Hart, 2002) suggested that 
participation in teacher education programs could influence preservice teachers’ beliefs. In teacher education 
courses, teacher educators create learning environments through specific teaching strategies that allow preservice 
teachers to experience models of teaching and learning that they might use in their future teaching. Supporting the 
strategies that teacher educators employ are their underlying beliefs about teaching and learning (Pajares, 1992). 
Since we may be able to predict teacher educators’ teaching practices from their beliefs, and their practices are likely 
to influence preservice teachers’ beliefs, it may be possible to predict preservice teachers’ beliefs from teacher 
educators’ beliefs. Thus, one of the research interests in this study was the relationship between teacher educators’ 
beliefs and preservice teachers’ beliefs. 

Besides beliefs, attitudes are thought to serve as predictors of behaviors (Richardson, 2003). Applied to 
technology use, attitudes toward technology are expected to predict one’s uses of technology. According to Aiken 
(1980), attitudes can be “learned directly by observing the activities of people who are perceived as significanf’ (p. 
16). In teacher education programs, teacher educators serve as models for preservice teachers. Therefore, to better 
prepare technology-using teachers, teacher educators need to model technology use by integrating technology into 
their classes (Parker, 1997; Vannatta, 2000; Willis & Tucker, 2001). As Faison (1996) commented, “Teacher 
preparation programs must have delineated goals for technology infusion and provide consistent modeling of 
effective uses of technology in the classroom and in curriculum” (p. 58). 

Abbott and Paris (2000) examined the attitudes of elementary education preservice teachers toward the use 
of computers before and after a semester-long site-based literacy course that integrated technology to support 
preservice teachers’ understandings of elementary teaching. The participants’ responses to the attitude surveys 
revealed an increase in positive attitudes toward computers after taking this course. In social studies education 
courses, Crowe (2004) modeled technology use and had prospective teachers work on technology-integrated 
projects. Most of the students felt that “the instructor’s use of technology made them feel more comfortable that they 
could use technology in their teaching and that they had gotten ideas about how to use technology to enhance 
student learning from the activities in class” (p. 164). One student’s comments specified, “the instructors’ modeling 
- teaching with technology - was important for her to decide that she could teach with technology” (p. 164). It can 
be seen that preservice teachers’ positive attitudes can be developed through their experiences in classes in which 
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teacher educators used computer technology to facilitate teaching and learning. Thus, one question arises: can 
teacher educators’ technology uses predict preservice teachers’ technology attitudes? To answer this question, we 
explored the relationship between teacher educators’ technology uses and preservice teachers’ technology attitudes. 



Research Questions 

The purpose of this study was to examine preservice teachers’ pedagogical beliefs and attitudes toward 
technology in relation to teacher educators’ pedagogical beliefs and their technology uses. Specifically the study 
intended to answer the following questions: 

1 . What is the relationship between teacher educators’ beliefs and preservice teachers’ beliefs? 

2. What is the relationship between teacher educators’ uses of technology and preservice teachers’ 
attitudes toward technology? 



Methods 

To examine the research questions, a correlational research design was employed. Multiple regression 
techniques were used. 

Participants and Site 

The study was conducted with instructors and new teacher education students in a College of Education at a 
large mid- western university in spring 2005. The participants consisted of 96 new teacher education students from 
1 1 teaching majors and 14 instructors from 3 teacher education courses, referred to here as A, B, and C. Specifically, 
six instructors were teaching course A, seven instructors were teaching course B, and one instructor was teaching 
course C. Course A and course B introduced students to general educational theory and practice. Students in these 
two courses formed a cohort, with all those attending one section of course A also taking the corresponding section 
of course B. Course C was an introductory educational technology course. All student participants took course A 
and course B. A total of 75 of the 96 students also took course C. The three courses were the only teacher education 
courses that these students took in the semester. 

Data Collection 

Data were collected through online surveys. At the beginning of the semester, a pre-survey was 
administered to the student participants to assess their current beliefs and attitudes toward technology. At the end of 
the semester, a post-survey was completed by the student participants to examine their beliefs and technology 
attitudes. The post-survey was similar to the pre-survey except that there were two open-ended questions asking 
students about their perceptions of the influence of their teacher education courses on their beliefs and technology 
attitudes. At the end of the semester, a survey was administered to instructor participants to examine their beliefs 
and technology uses in instruction. 

Instruments 

Teacher educators and preservice teachers’ beliefs were measured by the Teacher Beliefs Survey 
(McCombs & Whisler, 1997). This survey contains 35 rating scale items. Three factors were identified: (a) learner- 
centered beliefs about learners, learning, and teaching (LB); (b) non-leamer-centered beliefs about learners (NLB- 
L); and (c) non-leamer-centered beliefs about learning and teaching (NLB-LT). 

Preservice teachers’ attitudes toward technology were measured by a questionnaire developed by Francis- 
Pelton and Pelton (1996). An exploratory factor analysis was conducted which yielded 4 factors (33 items). The 
researchers labeled the four factors as: educational benefits (EB), confidence (CF), impact on teaching (IT), and 
concerns about effects on students (CS). 

Teacher educators’ technology uses in instmction were measured through two sections of the survey. The 
first part of the survey measured the frequency of using a variety of tools and applications using questions from the 
Computer Tools/ Applications section of a faculty survey (Lehman & Brown, 2001). The second part of the survey 
measured the ways in which the instmctors required the student participants to use computer technology. This 
section contained 12 items that fell into two categories. One category included 8 items adapted from the survey 
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items in the studies conducted by Becker (2000) and Niederhauser and Stoddart (2001). This category measured the 
frequency with which instructors required students to use technology in constructivist ways. The other category 
contained 4 items adapted from the survey items in Becker’s study (2000), which measured the frequency with 
which instructors required students to use technology in traditional ways. 

Two open-ended questions were included at the end of the teacher educators’ survey. The instructor 
participants were asked to state how they thought the students’ teaching philosophies and technology attitudes might 
have developed as a result of taking their courses. Two open-ended questions were included at the end of the 
preservice teachers’ post- survey. The student participants were asked to reflect on how the teacher education courses 
they took in the semester influenced their teaching philosophies and their attitudes toward technology. 

Data Analysis 

Descriptive data from the pre-survey and the post- survey were calculated for both students and instructors. 
To score students’ attitudes toward technology, items stated negatively were coded with 1 indicating negative 
attitudes and 5 indicating positive attitudes. MANOVA (multivariate analysis of variance) was used to examine 
whether demographic data would make a difference in student participants’ beliefs and technology attitudes prior to 
enrolling in the initial teacher education courses. The effect size of the multivariate T|^ is reported. In addition, t tests 
were conducted to see whether there was a significant change in students’ beliefs and technology attitudes from pre- 
survey to post-survey. An effect size d was calculated. Besides three beliefs scores, data reported for instructors 
included the use of various computer tools and the ways in which teacher educators required the student participants 
to use technology. MANOVA tests were performed to examine whether the instructor participants’ beliefs and 
technology uses were different across the three courses. 

When testing the relationship between teacher educators’ beliefs and preservice teachers’ beliefs, sequential 
multiple regression analysis was conducted to examine each belief subscale separately. The students’ post- survey 
beliefs scores were regressed as functions of instructors’ beliefs scores. To control for the influence of student 
participants’ pre-existing beliefs, their beliefs scores on the pre-survey were used as covariates in the regression 
analyses. Sequential multiple regression analyses were also performed when testing the relationship between teacher 
educators’ technology uses and preservice teachers’ technology attitudes. Students’ post-survey attitudes scores 
were regressed as functions of instructors’ technology use scores, and the students’ pre-survey attitudes scores were 
used as the covariate. Since not all the student participants took course C, a dummy variable was used to indicate 
course C instructor’s data in each multiple regression analysis. 

When analyzing the participants’ responses to open-ended questions, individual responses were examined. 
Responses reflecting identical or similar ideas were combined into one category. In this way, different categories 
were developed to demonstrate different perceptions on a particular question. 

Results 

Descriptive statistics of student participants’ data are presented in Table 1. MANOVA tests results 
suggested no significant differences in beliefs scores {p = .42, = 0.31) or technology attitudes scores {p = .88, 

= 0.30) on the pre-survey among the students who were enrolled in different teaching majors. 



Table 1 

Descriptive Statistics of Students ’ Data {N — 96) 









Students’ Beliefs 


Students’ Technology Attitudes 


LB 


NLB-L 


NLB-LT 


EB 


CF 


IT 


CS 


Pre-survey 


M 


3.28 


2.08 


2.79 


3.70 


4.07 


4.34 


3.74 




SD 


0.29 


0.37 


0.37 


0.40 


0.71 


0.41 


0.58 


Post-survey 


M 


3.30 


2.16 


2.64 


3.86 


4.15 


4.30 


3.74 




SD 


0.34 


0.45 


0.40 


0.56 


0.58 


0.53 


0.70 



Note. LB=leamer centered beliefs; NLB-L=non-leamer-centered beliefs about learners; NLB-LT=non-leamer-centered beliefs 
about learning and teaching; EB=educational benefits; CF=confidence; IT=impact on teaching; CS=concems about effects on 
students. 



T-tests revealed that students’ scores on non-leamer-centered beliefs about learners on the pre-survey were 
significantly lower than their scores on the post-survey; ^[95] = 2.39, p = .02, d= .3. Students’ non-leamer-centered 
beliefs about learning and teaching on the pre-survey were significantly higher than on the post-survey; ^[95] = - 
4.29,/7<.0001,r/=.4. 
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When comparing those who took course C to those who did not, no significant difference was found among 
these groups on their pre-survey technology attitudes scores. On the post-survey, however, the technology attitudes 
scores, specifically related to educational benefits, for those who took course C were significantly higher than for 
those who did not take course C (^93] = 2.02, p = .025 [one-tailed], d= .5), after deleting one outlier. 

The instructor participants’ data are presented in Table 2. MANOVA results indicated that they were not 
significantly different in their beliefs (p = .47, r\^ = 0.24) and technology uses (p = .76,r\^ = 0.35) across the three 
courses. 

Table 2 



Instructors’ Beliefs and Technology Uses across Three Courses (N= 14) 







M 


SD 




LB 


3.28 


0.41 


Instmctors’ beliefs 


NLB-L 


2 


0.47 




NLB-LT 


2.20 


0.43 




Frequency of using various tools 


1.98 


0.27 


Instmctors’ technology 
uses 


Frequency of requiring students to use 
technology in constmctivist ways 


2.94 


0.58 




Frequency of requiring students to use 
technology in traditional ways 


2.07 


0.73 



Note. LB=leamer centered beliefs; NLB-L=non-leamer-centered beliefs about learners; NLB-LT=non-leamer-centered beliefs 
about learning and teaching. 

Relationship Between Teacher Educators’ Beliefs and Preservice Teachers’ Beliefs 

The regression analyses revealed that the learner-centered beliefs of the course A instructors were 
statistically significant (p = .003) in predicting students’ learner-centered beliefs (see Table 3). The squared semi- 
partial correlation of .06 indicated that it explained 6% of the variance (sr^= .06) in the students’ post-survey scores 
on learner-centered beliefs. As the covariate, the students’ pre-survey scores on learner-centered beliefs explained 
35% of the variance in the students’ post-survey scores. 

Table 3 



Results of Regression Analysis of Learner-Centered Beliefs 



Variable 


B 


R^ 


F 


P 


P 


2 

sr 


Step 1 

Students’ LB on pre-survey 


0.59 


.34 


47.57 


<.0001* 


.59 


.35 


Step 2 

Course A instmctors’ LB 


0.32 


.40 


9.49 


.003* 


.25 


.06 


Step 3 

Course C instmctor 


-.04 


.403 


0.35 


.56 






Step 4 

Course B instmctors’ LB 


.01 


.403 


0.04 


.84 







Note. 5=raw regression coefficient; p=standardized regression coefficient; ^r^=squared semi-partial correlation. 
*/><.05 



No statistical significant results were found in regression analysis of non-leamer-centered beliefs about 
learners. Table 4 reports the results of the regression analysis of non-leamer-centered beliefs about learning and 
teaching. The non-leamer-centered beliefs about learning and teaching of course A instmctors were also statistically 
significant {p = .02) in predicting students’ non-leamer-centered beliefs about learning and teaching and explained 
4% of the variance in students’ post-survey scores {sr^ = .04). As the covariate, the students’ non-leamer-centered 
beliefs about learning and teaching were significant {p < .0001) and explained 44% of the variance in students’ non- 
leamer-centered beliefs about learning and teaching on the post- survey {sr^ = .44). 
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Table 4 



Results of Regression Analysis of Non-Learner-Centered Beliefs About Learning and Teaching 



Variable 


B 


R^ 


F 


P 


p 


2 

sr 


Step 1 

Students’ NLB-LT on pre-survey 


0.69 


.43 


68.72 


<.0001* 


.67 


.44 


Step 2 

Course A instructors’ NLB-LT 


0.25 


.46 


6.15 


.02* 


.19 


.04 


Step 3 

Course C instructor 


0.07 


.47 


.84 


.36 






Step 4 

Course B instructors’ NLB-LT 


-0.02 


.47 


.07 


.80 







Note. 5=raw regression eoeffieient; p=standardized regression eoeffieient; ^r^=squared semi-partial eorrelation. 






According to the students’ responses to the open-ended questions, students perceived that all three courses 
influenced their teaching philosophies in one way or another. The majority {n = 67) of the student participants 
reflected that taking course A positively influenced their teaching philosophies. This course gave them insights into 
teaching and learning, and helped them learn how to be a teacher. This finding helped to understand the quantitative 
findings that course A instructors’ learner-centered beliefs and non-leamer-centered beliefs about learning and 
teaching were significant in predicting students’ beliefs. 

Relationship Between Teacher Educators’ Technology Uses and Preservice Teachers’ Technology Attitudes 

A significant predictive relationship was found between teacher educators’ technology uses and preservice 
teachers’ technology attitudes related to education benefits. Since the variable of “frequency that instructors had 
students use technology in constructivist ways” was highly correlated to the variable of “frequency that instructors 
had students use technology in traditional ways”, the two variables were included in separate regression models. The 
results are presented in Table 5 and Table 6. 

Table 5 

Results of Regression Model 1 in Predicting Students ’ Technology Attitudes Related to Education Benefits 



Variable 


B 


R^ 


F 


P 


P 


s? 


Step 1 

Students’ attitudes on pre-survey 


0.11 


.38 


55.91 


.0006* 


.62 


.38 


Step 2 

Course C instructor 


0.28 


.44 


9.24 


.016* 


.21 


.04 


Step 3 

Use of tools by course A instructors 


-0.27 


.46 


4.05 


.16 






Step 4 

Constructivist use by course A instructors 


0.14 


.47 


1.98 


.40 






Step 5 

Constructivist use by course B instructors 


-0.14 


.48 


1.71 


.44 






Step 6 

Use of tools by course B instructors 


-0.14 


.48 


0.36 


.81 







Note. 5=raw regression eoeffieient; p=standardized regression eoeffieient; ^r^=squared semi-partial eorrelation. 
*/><.05 
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Table 6 

Results of Regression Model 2 in Predicting Students ' Technology Attitudes Related to Education Benefits 



Variable 


B 


R^ 


F 


P 


p 


2 

sr 


Step 1 

Students’ attitudes on pre-survey 
Step 2 


0.77 


.38 


55.91 


.0006* 


.62 


.38 


Course C instructor 


0.28 


.44 


9.24 


.016* 


.21 


.04 


Step 3 

Use of tools by course A instructors 


-0.27 


.46 


4.05 


.16 






Step 4 

Use of tools by course B instructors 


-0.26 


.47 


1.44 


.50 






Step 5 

Traditional use by course A instructors 


0.11 


.All 


0.69 


.72 






Step 6 

Traditional use by course B instructors 


-0.02 


.All 


0.17 


.90 







Note. 5=raw regression eoeffieient; p=standardized regression eoeffieient; ^r^=squared semi-partial eorrelation. 






As the covariate, it is not surprising to find that the students’ attitude scores on the pre-survey were the 
strongest predictor of their attitude scores on the post-survey. The dummy variable representing course C instructor 
was significant (/? = .016) in predicting students’ technology attitudes related to the educational benefits. That is, 
taking this course had a positive relationship to students’ technology attitudes (P = .21) and explained 4% of the 
variance {sr^= .04) in the students’ post-survey scores. No significant relationship was found between the 
instructors’ technology uses and the students’ attitudes related to confidence, impact on teaching, or concerns about 
effects on students. 

The analysis of the student participants’ responses to the open-ended question showed that 55 out of 75 
students who took course C thought that this course had influenced their attitudes toward technology. This course 
taught them how to integrate computer technology into classes and helped them realize the importance and 
usefulness of technology in education. Few of them mentioned courses A or B. In response to the open-ended 
questions, course C instructor thought the overall influence of this course was to help the students see the power of 
technology in facilitating teaching and learning. In general, the instructors in courses A and B did not think the 
students’ technology attitudes were influenced by their courses since the students were not exposed to technology a 
lot in the classes. This supports the quantitative findings that no positive relationship was found between the 
instructors’ technology uses in course A and course B and students’ technology attitudes. 

Discussion and Implications 



Beliefs 



An important goal of teacher education programs is to help preservice teachers identify and develop their 
beliefs in relation to teaching. In teacher education programs, preservice teachers’ entering beliefs may be 
“challenged through classroom readings, dialogue, and classroom experimentation” (Richardson, 2003, p. 1). By 
means of these activities, it is expected that teacher educators can share their perceptions and ideas about teaching 
and learning with the students. This study revealed that the instructors’ learner-centered beliefs and non-leamer- 
centered beliefs about learning and teaching had an influence on the students’ beliefs, however, the influence was 
small over the course of one semester. According to Richardson (2003), preservice teachers’ entering beliefs are 
usually deep-seated. This is supported by the findings in this study in which students’ beliefs scores on the pre- 
survey were always the strongest predictors of their scores on the post- survey. Expecting teacher educators’ beliefs 
to have a great influence on preservice teachers’ beliefs over a single semester may have been unrealistic. Future 
research should examine the predictive relationship between teacher educators’ beliefs and preservice teachers’ 
beliefs over a longer period of time. 

Interestingly, the results of this study indicated that only course A instructors’ beliefs were significant in 
predicting students’ beliefs. Furthermore, the majority of the student participants noted that taking course A 
positively influenced their teaching philosophies and helped them learn how to be teachers. It is possible that course 
A instructors had different beliefs than the other instructors and that these differences may have manifested 
themselves in their classroom activities. Alternatively, it is possible that these instructors simply discussed their 
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beliefs more readily, and engaged their students in activities that allowed them to reflect on their own beliefs and 
ideas. 

Although it is hard to change deep-seated beliefs, such as those that preservice teachers may bring with 
them to their teacher education programs, this is not impossible, which has been indicated by some studies 
(Cifuentes, 1997; Daniel, 1996; Friel & Carboni, 2000; Hart, 2002). In addition, it is likely that these changes do not 
occur evenly over time, but rather in starts and stops as existing beliefs are subjected to deeper reflection, such as 
when they are challenged by new information or opposing beliefs. For example, in this study, preservice teachers’ 
non-leamer-centered beliefs about learning and teaching were significantly weaker at the end of the semester than at 
the beginning, however, their non-leamer-centered beliefs about learners were significantly stronger at the end of the 
semester. This suggests that as preservice teachers decreased their beliefs about teacher control, they tended to 
become more concerned about students’ ability to succeed in the learning process. In practice, teacher educators 
may need to be aware of the possible difference in the development of preservice teachers’ non-leamer-centered 
beliefs and discover ways to address them simultaneously. According to McCombs and Whisler (1997): 

We found that when teachers’ beliefs about learners were non-leamer centered (that is, when teachers did 
not believe that all students can be helped to learn), then students tended to perceive classroom practices in 
negative ways. And when student perceptions were negative, both student motivation and classroom 
achievement were low (p. 199). 

Thus, teacher educators need to pay particular attention to the development of preservice teachers’ non-leamer- 
centered beliefs about learners. As teacher educators help preservice teachers understand students’ roles in the 
teaching and learning process, it is very important to help them also realize the uniqueness of each learner and to 
understand how to support students’ learning. 

Significant changes in preservice teachers’ beliefs may require long-term efforts throughout the entire 
teacher education program. It is expected that teacher educators could engage students in learning activities that 
allow them to experience, to reflect, and then to modify their beliefs. Teacher educators’ guidance is important. In 
this process, hopefully, teacher educators’ beliefs could be projected through teaching practices to influence 
students’ beliefs. 

Technology Attitudes 

Research (Abbott & Faris, 2000; Crowe, 2004; Kumar & Kumar, 2003) suggested that preservice teachers’ 
attitudes toward technology may be improved if teacher educators integrated technology into teacher education 
course work. In this study, it was found that teacher educators’ technology uses, in terms of using computer 
tools/applications and requiring students to use technology in different ways, did not predict students’ technology 
attitudes. However, taking course C, an introductory educational technology course, was found to be statistically 
significant in predicting students’ technology attitudes related to educational benefits. Unfortunately, this cannot be 
directly related to course C instmctor’s technology uses, but rather to taking a course that provided relevant 
information about how and why to use a variety of technologies in the classroom as well as a range of hands-on 
experiences. 

Although the results of this study did not do justice to the assumption that teacher educators’ technology 
uses could predict preservice teachers’ technology attitudes, one should not jump to the conclusion that preservice 
teachers’ technology attitudes are not related to teacher educators’ technology uses. According to the comments 
made by the instmctors in course A and course B, the students were not exposed to much technology in their 
courses, therefore, they did not expect the students’ attitudes toward technology to have been influenced. Thus, one 
possible explanation for the lack of positive relationship between the instmctors’ technology uses and the students’ 
attitudes may be that the instmctors’ technology uses were not at a level that could be influential. Although 
instmctors had a certain amount of technology use, such use may not have been enough to influence the students’ 
technology attitudes. To better explore this relationship in the future, data could be collected from new teacher 
education students and teacher educators who use more technology and use them at a higher level. 

The regression results revealed that taking course C made a small contribution to the prediction of students’ 
attitudes over one semester. Additionally, the t-tests results indicated that taking this introductory educational 
technology course was helpful for improving preservice teachers’ technology attitudes related to educational 
benefits. While the student participants tended to have positive technology attitudes related to confidence and impact 
on teaching before they entered the teacher education program, their attitudes related to educational benefits and 
concerns about effects on students were not very positive. Furthermore, at the start of the study, those who took an 
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introductory educational technology course and those who did not take the course did not significantly differ in their 
attitudes related to educational benefits. However, at the end of the semester, those who took the course had 
significantly stronger attitudes than those who did not take the course. The students’ comments on the influence of 
course C on their attitudes mainly centered upon the usefulness and importance of technology in education. This 
suggests that in teacher education programs, it may be important for students to take an educational technology 
course that presents ideas about how to integrate technology. Such an introductory course will not only provide 
students with foundational knowledge and skills related to educational technology, but also facilitate their positive 
attitudes toward technology use in education. This will increase the possibility that preservice teachers will use 
technology in their future teaching practice (Myers & Halpin, 2002; Yildirim, 2000). 

Preservice teachers need to witness and experience technology use in a pedagogically sound manner as 
students, and then hopefully they will implement that in their own teaching. Facilitating preservice teachers’ positive 
attitudes toward technology should not only be the responsibility of teacher educators in technology courses. It 
requires the joint efforts of all teacher educators. 

Limitations and Recommendations for Future Research 

According to Richardson (1996), the measurement of teachers’ beliefs in the form of multiple-choice is 
“too constraining” (p. 107). Richardson further commented that multiple-choice measures of beliefs that are 
predetermined by the researchers “often do not validly represent teachers’ beliefs” (p. 107). Due to the design of this 
study, a multiple-choice test was selected to measure the beliefs. However, it is recommended that, if possible, a 
qualitative study be designed to explore the relationship between teacher educators’ beliefs and preservice teachers’ 
beliefs. Hopefully, a qualitative approach will provide rich and in-depth exploration and a better description of the 
relationship. 

In the future, when time and funding permits, a longitudinal study could be conducted to examine the 
development of preservice teachers’ beliefs over the course of their teacher education programs. Since preservice 
teachers usually have strong and deep-seated beliefs when they enter the teacher education program, a study on the 
development of their beliefs over a long period could provide more useful information for practitioners and 
researchers. 

A small sample size may have influenced the results of this study. In the future, when conducting similar 
studies, a large sample size is recommended. Finally, as discussed earlier, one possible explanation for the lack of a 
predictive relationship between teacher educators’ technology uses and preservice teachers’ technology attitudes 
may be that the instructors’ technology uses were not at a level that could be influential. To better examine this 
issue, it is recommended that a comparative study be conducted to examine the difference of influence on preservice 
teachers’ technology attitudes between those teacher educators who expose students to more technology experiences 
and those who do not expose students to a lot of technology experiences. 

Conclusion 

Preservice teachers’ beliefs and technology attitudes are two important issues in teacher education 
programs. This study examined these issues in relation to teacher educators’ beliefs and technology attitudes. The 
results were reported and discussed, which also provided avenues for future studies. It is expected that such 
exploration would enrich our knowledge about how to prepare technology-using teachers and suggest good practices 
for teacher educators. 
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Cognitive Load in Online Reading 

Shujen L. Chang 
University of Houston 
Yegmin Chang 

National Cheng-Chi University 
Abstract 

This study investigated the effects of different material formats (paper, online- 
linear, and online-nonlinear) on reading comprehension and time needed to complete 
reading from a cognitive load perspective. One hundred and thirty five undergraduate 
students participated in the experiment. Significant differences of reading comprehension 
and time were found among groups using different material formats, while participants 
reported no significant differences in content and format cognitive loads among groups. 

Introduction 

Educators have noted that online reading adversely influenced reading 
comprehension (Patterson, 2000; Shapiro, 1998; Shapiro & Niederhauser, 2004). This 
influence may be attributed to the format of online material and cognitive load required 
for reading online material. The format of online material was changed from traditional 
linear paper format to nonlinear multi-window format (Barker & Tedd, 1999; Quinlan, 
1997; Wiley & Schooler, 2001); reading nonlinear multi-window material might require 
greater cognitive load (Britt & Gabrys, 2001; Hill & Hannafm, 2001; Yang, 2001). 
However, is online reading comprehension affected by the nonlinearity of online material 
formats or by reading from a digital display on screen instead of paper? Do different 
material formats require a change in cognitive load and therefore need various lengths of 
time to complete reading? Does the content or the format of reading material cause 
increase in cognitive load? These questions have not been empirically inspected so far. 

This study investigated the effects of the different material formats (paper, online- 
linear, and online-nonlinear) on reading comprehension and the length of time needed to 
complete reading from a cognitive load perspective. Reading comprehension referred to 
the score a participant earned from the reading tests. The length of time referred to how 
many minutes a participant used to complete reading and tests. The content and the 
format cognitive loads were participants’ report of the extent of perceived mental effort 
spent with respect to the content or the format of reading material. The research question 
examined whether reading different material formats would require various levels of the 
content and the format cognitive loads, which would subsequently interfere reading 
comprehension and require more time to complete the readings. 

Method 

Design 

One hundred and thirty five undergraduate students (95 females and 35 males) of 
College of Commerce at a large state university in Taiwan participated in this study 
voluntarily. This study was a three-group randomized experimental design. Each group 
received the same reading material but in a different format. This study observed two 
dependent variables, reading comprehension and time, with two covariates, content and 
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format cognitive loads under the treatment - three different material formats: paper, 
online-linear, and online-nonlinear. 

Reading Material 

The reading material ineluded three short reading passages in Chinese and eaeh passage 
was followed by a test eontaining 5 multiple-choice questions. All groups received the 
same eontent of the reading material but in different formats. The paper format linearly 
displayed passages and tests on paper; although linearly, the online-linear format 
presented passages and tests on computer sereen within a single- window; and the online- 
nonlinear format displayed a list of three links on sereen and each link opened a new 
window displaying the corresponding passage and test. All groups were asked to reeord 
answers of tests on the given answer sheet. 

Variables 

The two dependent variables were reading eomprehension and time used to 
complete reading and tests. Reading eomprehension was measured by the total seore that 
a participant earned from the three reading tests; the time was the number of minutes 
used to complete reading and tests reported by partieipants. 

The two eovariates were eontent and format cognitive loads referred to 
partieipants’ report of the extent of self-perceived mental effort invested in reading 
comprehension with respect to the content and the format of reading material. Content 
and format eognitive loads were measured via two cognitive load questions on a 
subjeetive 9-point rating seale developed by Paas (1992). 

Proeedure 

The researehers, first, explained the purpose of this study and randomly assigned 
partieipants into three groups: the paper group in a regular elassroom and the two online 
groups in a eomputer laboratory with Internet eonnection. Second, the researchers 
distributed and explained instruetion with the answer sheet. Finally, the researehers 
collected answer sheets after partieipants finished the experiment. 

Data Analysis 

MANCOVA tests with eontent and format eognitive loads as two interactive 
covariates were used to determine whether there were differenees concerning the effects 
of different material formats on reading comprehension and time. MANOVA tests were 
also employed to determine whether different formats required different amount of 
content and format cognitive loads. Seatter plots and Pearson eorrelations were used to 
determine the assoeiation between reading comprehension and time. A priori of .05 
significance level was used for all statistical tests in this study. 

Results 

Reading comprehension and time had means 1 1.75 points (SD = 1.90) ranging 
from 5 to 15, and 13.96 minutes (SD = 2.93) ranging from 5 to 15 minutes. Content and 
format eognitive load had means 14.80 (SD = 4.01) and 14.16 (SD = 4.31) on a 9-point 
scale, respectively. 

MANCOVA tests indieated significant differences among groups in reading 
comprehension and time while content and format eognitive load were used as eovariates 
(Pillais’ Trace p = .01). Tests of between-subjeet effeets also indicated significant 
differences among groups in reading comprehension (p = .01) and time (p = .05). On 
reading comprehension, the paper group achieved the highest score (M= 12.24), the online- 
nonlinear group aehieved the medium seore (M= 1 1.74), and the online- linear group 
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achieved the lowest score {M= 1 1.28). As to time, the paper group used the shortest time {M 
= 13.24), the online-linear group used the medium time (M= 13.65), and the online- 
nonlinear group used the longest time {M= 15.05). However, scatter plots and Pearson 
correlations indicated no significant correlation between reading comprehension and time 
{r=m,p = .9A). 

MANOVA tests indicated no significant differences among groups concerning 
content and format cognitive loads (Pillais’ Trace p = .62, Partial Eta Squared = .01, 
Observed Power = .21). Also, tests of between-subject effects indicated no significant 
differences among groups concerning content cognitive load {F = .60, p = .55) and format 
cognitive load {F= .\\,p = .90). 

Discussion 

Reading Comprehension 

Significant differences of reading comprehension were found among groups using 
different formats of reading. Specifically, the paper group outperformed both online-linear 
and online-nonlinear groups on reading comprehension. The finding confirms with 
previous findings that online reading does interfere reading comprehension (Patterson, 
2000; Shapiro, 1998; Shapiro & Niederhauser, 2004). However, the finding that online- 
nonlinear group outperformed the online-linear group on reading comprehension is not in 
line with previous studies (Britt & Gabrys, 2001; Hill & Hannafm, 2001; Yang, 2001). 
This finding implies that online-linear material may not facilitate reading comprehension. 
Specifically, scrolling up and down on screen to read online linear material may interfere 
reading comprehension the most, compared to flipping through paper or clicking links. 

Time 

Consistent with our hypotheses, the time used to complete reading and tests was 
found significantly different among groups. Specifically, the paper group used the shortest 
time, the online-linear group used the medium time, and the online-nonlinear group used the 
longest time. This finding indicates that reading online, compared to paper, material takes 
more time because of the required additional activities for online reading, such as scrolling 
on screen and navigating multi-windows. Also, navigating between multi-windows may 
take more time than scrolling on screen because additional time is needed to access multi- 
windows. Thus, online instruction should allow more time for online reading. In addition, it 
was found that there was no significant correlation between time and reading 
comprehension. This finding suggests that the additional time needed for online learning 
may not affect online learning effectiveness. 

Content and Format Cognitive Loads 

Participants using different formats reported no significant differences in perceived 
content and format cognitive loads, contrary to previous studies (Britt & Gabrys, 2001; 

Hill & Hannafm, 2001; Yang, 2001). The short length of reading passages in this study 
may not create significant variations in content and format cognitive loads among groups. 

In summary, this study found that reading comprehension and time needed for 
completing reading were significantly affected by material formats (paper, online-linear, 
and online-nonlinear). However, participants using different formats reported no 
significant differences of perceived content and format cognitive loads. 
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Investigating the Use of Advance Organizers as an Instructional Strategy for 

Web-based Distance Education 



Baiyun Chen, Atsusi Hirumi 
University of Central Florida 

Abstract 

It is synthesized that advance organizers (AOs), an effective orienting device in traditional classroom instruction, 
may enhance students’ information literacy in self-directed online classes. The current study investigated the 
short-term and long-term effects of two types of advance organizers, graphic and text, in a fully Web-based 
undergraduate course on health care ethics. Although the results failed to yield a statistically significant difference 
regarding the use of AOs among treatment groups and the control group, additional qualitative data indicated that 
students held overwhelmingly positive attitudes towards using AOs, especially the concept map, in online learning. 
The analyses and results of this study added new empirical evidence for the use of AOs in Web-based distance 
education and posited new directions for further research. 

In fully Web-based courses, the use of hyperlinked multimedia resources often bring challenges of 
cognitive overload and learner disorientation (Dias & Sousa, 1997). While the learners enjoy the flexibility and 
abundance of Internet resources, they may also be overwhelmed with multiple tasks and sources of information. As 
distant learners, the students cannot always obtain immediate feedback from instructors, and need to make decisions 
on their own to locate course materials and complete assignments. While they recognize hyperlinks as a method for 
accessing electronic resources and navigating online courses, these students are likely to experience greater 
difficulties than students in instructor-led, face-to-face classes in terms of navigation and guidance. To better exploit 
the capabilities of technology, the adoption of effective online teaching and learning strategies is suggested to 
resolve online learning challenges (Bonk & Dennen, 2003). 

What kind of teaching and learning strategy exists for effective Web-based learning? Unfortunately, there 
is limited research to prove the effectiveness of learning strategies in fully Web-based environments. While many 
studies have shown no significant difference between online courses and traditional courses, applying traditional 
learning strategies at a distance often causes frustrations (Howell, Williams, & Lindsay, 2003). The literature reveals 
an increasing need to exploit research-based pedagogical strategies in fully Web-based environments. 

Distance Education 

Online learning has become pervasive in higher education. According to the latest survey, nearly two-thirds of 
all colleges and universities that deliver face-to-face instruction now offer online instruction, and the enrollment in 
2004 in Web-based courses reached 2.35 million, up nearly 20% over the 2003 figures (Allen & Seaman, 2004, 
2005). 

Both students and instructors perceive that online learning provides major benefits. The biggest benefit is 
convenience. Students can access learning materials at virtually anytime and almost any place. Also, the Internet 
brings expanded resources to the learners at a very low cost. Recently, multi-media learning environments have been 
created with audio, video, graphic and animation to simulate true tasks which bring tremendous interests and 
motivation to learners. For the instructor, online teaching can also be beneficial in terms of structure and time, 
increased student outreach and contact, personal satisfaction, availability of expanded research tools, improved 
course management, and the ability to learn new technologies (Hartman, Dziuban, & Moskal, 2000). 

At the University of Central Florida (UCF), online courses have become one of the major learning modalities. 
In the academic year of 2004-2005, 29,187 students, accounted for 65% of the total student population, were 
enrolled in the fully web-based classes (Center for Distributed Learning, 2006). Every semester, more than 180 
online courses have been offered through WebCT, and currently there are 6 undergraduate programs online, 10 
graduate programs online, and 10 graduate certificate programs online are available online. Although the Web has 
become an important instructional delivery mechanism, many students and faculty still fear self-directed learning 
processes and worry about the quality of online courses. Application of appropriate online instructional strategies is 
suggested to help students master course contents and to improve the quality and effectiveness of online courses. 

Purpose 

The current study investigated short-term and long-term effects of two kinds of AOs in a fully Web-based 
course. A concept map was used as a visual organizer, and an outline was used as a text organizer. Students’ 
knowledge acquisition and application were tested both immediately and four weeks after the instruction. All course 
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materials and assignments were accessed and completed on the Internet without any face-to-face instruction or 
meetings. 

This study strengthens the connections between theory and educational practice by providing empirical 
evidence for the use of AOs in distance education based on cognitive learning theories. 

Specifically, this study is designed to explore one instructional strategy, the use of AOs, to promote 
information literacy in a fully web-based distance course. Two hypotheses were posited for this study. 

Null hypothesis /. There is no difference in the short-term knowledge-based and performance-based learning 
achievements among students in the concept map, outline and control groups. 

Null hypothesis II. There is no difference in the long-term knowledge-based and performance-based learning 
achievements among students in the concept map, outline and control groups. 

Foundations 

Theoretical Foundation 

The rationale for using advance organizers (AOs) is rooted in cognitive learning theories. Cognitive theories 
claim that learning depends on processing capacity and prior knowledge (Driscoll, 1999). With the aid of AOs, 
learners are able to link what they already know to new information and apply it to new context. 

Ausubel (1968) first introduced the concept of AOs in his assimilation theory of meaningful learning and 
retention. Like other cognitive theorists, Ausubel asserted that learning is based on schemata or mental structures by 
which students organize their perceived environment. Ausubel suggested that AOs help students activate prior 
knowledge in the new instructional context and make the instructional process meaningful to the students (Ausubel, 
2000). His early experiments provided the most-cited research supporting the effectiveness of AOs with increasing 
achievement (Ausubel, 1960; Ausubel & Fitzgerald, 1961, 1962; Ausubel & Youssef, 1963). 

Mayer reinterpreted AusubeTs use of AOs in terms of his assimilation encoding theory (Mayer, 1979a). He 
indicated that the successful use of AOs is highly influenced by the availability of an assimilative context in memory 
and the active use of knowledge during learning. He reported the results of a series of nine studies and also 
examined 27 AOs studies conducted by other researchers in the 1960s and 1970s (Mayer, 1979a, 1979b). The results 
supported his contention that AOs will facilitate learning in situations where learners do not possess a rich set of 
relevant past experiences and can actively integrate the AOs in the new context. 

Based on neurophysiological science, the recent brain-based learning research also supports the idea of 
meaningful learning and active processing. It is asserted that meaning is more important than information and active 
information processing is strongly connected to prior learning (Caine & Caine, 1991). The brain-based learning 
theories also imply that the teacher needs to prepare the students before a unit of study to attach new information to 
prior knowledge so the new information has something to “latch onto”(Jensen, 1996). In addition, empirical studies 
support many of the proposition regarding AOs and learning posed by various theories. 

Empirical Evidence 

Extensive research was conducted on the effectiveness of using AOs in classroom teaching from the 1960s to 
the 1990s. The research evidence concerning any facilitative effect of AOs upon learning and retention is variable, 
but positive in general. Although AusubeTs early experiments supported the effectiveness of AOs with significant 
increasing learning achievement (Ausubel, 1960; Ausubel & Fitzgerald, 1961, 1962; Ausubel & Youssef, 1963), 
later studies failed to show a consistent positive facilitative effect (Barnes & Clawson, 1975; Luiten, Ames, & 
Ackerson, 1980; Mayer, 1979b; Stone, 1983). The discrepancies regarding the effectiveness of AOs might result 
from inadequate construction of AOs or weak research procedures or control (Kenny, 1993; Luiten et al., 1980; 
Mayer, 1979b). 

In the 1990s and 2000s, AOs still remained an actively debated topic. Research on the traditional AOs 
drastically decreased in number possibly due to the non-statistical-significance of the research results. However, 
many researchers began to conduct studies on AOs in a variety of formats, such as visual AOs (DaRos & 
Onwuegbuzie, 1999; Herron, Hanley, & Cole, 1995; Hirumi & Bowers, 1991; Millet, 2000) and multimedia AOs 
(Calandra, Lang, & Barron, 2002; Hale, 2003; Minchin Jr., 2004; Tseng, Wang, Lin, & Hung, 2002; Yeh & 

Lehman, 2001). Consistent with the historical findings, recent research again failed to generate statistically 
significant results on effectiveness of AOs on posttest scores between treatment groups and control groups, though 
most researchers continued to suggest a mild but positive effect of AOs on learning and retention. 

The statistical non-significance of the research might be attributed to imprecise construction of organizers, 
short duration of treatment, inadequate research control, and insufficient instruction on how to use organizers 
(Kenny, 1993; Luiten et al., 1980; Mayer, 1979b). Synthesizing the findings of recent research on AOs, the current 
study is an attempt to: 
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1 . investigate AOs as a helpful orienting device not only in the context of computer-assisted instruction, but also 
in the online learning environment. 

2. test different types of AOs, including graphic, graphic + text, and multimedia instructional organizers. 

Method 

Subject 

Participants of this study were selected from a population of undergraduate students enrolled in a fully 
Web-based health class at the University of Central Florida. This course was a health related ethics course. It was 
required for all health-related major undergraduate students. The course was fully Web-based, requiring no 
face-to-face meetings, where the instructor and students logged into a WebCT account and communicated through 
course pages, discussions and e-mails within the online account. The course covered ethical issues in health care, 
including life-saving measures, rights to die, transplants, surrogate parenthood, privacy and confidentiality, and 
decision-making. 

The population encompassed 164 undergraduate students enrolled in a fully web-based class. The majority of 
the students were between the age of 21-23 in either their junior or senior year. Most students were health-related 
majors. Three fourths of the participants were female students, and most of the students are white Caucasians. All 
enrolled students were asked to participate in this study on a voluntary basis. Bonus points towards the course credit 
were awarded for participants who completed all instruments. Almost 80% of the class voluntarily participated in 
the study activities, and 63 of the 164 course participants completed both quizzes and a survey. 

Research Design 

This study used a repeated-measure, control-group posttest-only design with random assignment to examine 
the effects of advance organizers (AOs) on learning as illustrated in Figure 1. 



R 


E| 
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E2 


X 2 (Text Organizer) 
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R 
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(No Advance Organizer) 


05 


06 



Figure 1. Research Design Diagram 

“R” indicated that all participants were randomly assigned to three groups, two treatment groups (Ei and E 2 ) 
and one control group (C). AOs were the intervention in this experimental event. The experimental group (Ei) 
reviewed a concept map, a form of graphic organizer, before reading textbook. The comparison group (E 2 ) reviewed 
a text organizer, and the control group (C) did not read any AO before textbook reading. During the course of the 
study, all three groups completed an immediate posttest (Oi, O3, and O5) and a delayed posttest (O2, O4, and 05 ). 
Dependent & Independent Variables 

The dependent variable in this study is students’ learning achievement, encompassing their short-term (Oi, O3, 
and O5) and long-term knowledge acquisition and application (O2, O4, and 05 ). The short-term and long-term 
knowledge acquisition was tested with two corresponding 12-item knowledge quizzes. The short-term knowledge 
application was tested with problem-based scenario essay questions. 

The independent variable is the treatment of AOs (Xi & X2). The three groups had the same instruction, 
except for the treatment of AOs. The experimental group used a concept map (Xi); the comparison group used a 
textual outline (X2); and the control group had no AOs exposure before textbook reading. 

A posttest I-posttest II repeated-measure design was exploited in this study, and time was another independent 
variable for the research. It is assumed that the time factor might influence students’ learning achievement over a 
period of four weeks’ time. 

Advance Organizers 

Two forms of AOs were designed respectively for the experimental and comparison groups. The construction 
of the AOs was based on the criteria prescribed by Mayer (1979b) and followed a series of research-based 
procedures (Bricker, 1989; West, Farmer, & Wolff, 1991). Students were instructed to review the AOs before they 
read the textbook. Figure 2 and Figure 3 illustrate the AOs used in the study. 
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Figure 2. Graphic Organizer: Concept Map 
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Figure 2. Text Organizer: Textual Outline 



The graphic organizer is a flash-based interactive concept map as illustrated in Figure 2. The text organizer 
presents the same concepts and explanation as the concept map, as illustrated in Figure 3. Both AOs are linked to the 
instruction page of module 2. The only difference between the two organizers is the presentation of the relationship 
among the concepts. The concept map illustrates the relationship visually in a nonlinear way, and the textual outline 
presents it textually in a linear way. The validity of these AOs was tested and confirmed by expert review from both 
the instructor and the outside instructional designer, and modifications were made based on their suggestions. 

Procedures 

This study lasted for six weeks. During the first week of the spring semester, 2006, participants were 
randomly assigned into three groups. Each group was provided with one version of module 2 during week two. In 
the course module, the students were suggested to first review the advance organizers (AOs) to have an overall idea 
of the key concepts and issues covered in this module, if they had one available in their group. The students in the 
experimental group reviewed the multimedia concept map before reading the book. The students in the comparison 
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group reviewed the text outline before reading the book. The students in the control group were not given an AO, 
and they proceeded directly to textbook reading. 

Chapter 2, “The Physician-Patient Relationship” of the textbook, Biomedical Ethics (Mappes & Degrazia, 
2005), was the designated reading materials. After textbook reading, the students were instructed to complete all the 
assignments on the assignment page, including the two parts of the posttest I. The knowledge quiz of posttest I was a 
timed WebCT quiz. The students had 15 minutes to complete the 12 questions and they could only access and 
submit the quiz once. However, as this was a fully Web-based course, the quiz was not proctored and students had 
the flexibility to do the quiz at their convenience during the instruction week. For the second part of posttest II, the 
students completed three questions based on a scenario using Microsoft Word and submitted the assignment to the 
WebCT Dropbox tool by the next Monday morning. Also during this week, students filled out an online survey to 
report their background information and their uses of AOs. 

Four weeks after module 2, in week six, posttest II was administered through WebCT. Together with all the 
other assignments for module 6, posttest II, including a quiz and three scenario questions, was open for the students. 
It was stated in module 6 that both the quiz and scenario questions of posttest II were part of the voluntary research. 
The students completed this posttest with the knowledge they had learned in module 2. 

Research Findings 

Statistics procedures, including descriptive analysis, one-way analysis of variance (ANOVA), and 
repeated-measure regression were performed to study the research findings. 

Descriptive analysis was used for scores in the posttests. Means, standard deviations, and effect sizes of 
students’ achievement scores were computed for each quiz and scenario questions of posttest I and II. The 
assumptions of the analysis, including the homogeneity of variance and the normality of population distributions, 
were examined using the Fevene’s test and the Q-Q plot procedures. Detailed descriptive statistics of the students’ 
learning outcomes are illustrated in Table 1 and Table 3 in the next sections under the discussions of research 
hypotheses. 

Null Hypothesis I: There is no difference in the short-term knowledge-based and performance-based learning 
achievements among students in the concept map, outline and control groups. 

In posttest I, students of group I using a concept map had the highest mean score (36.25) in the knowledge 
quiz 1, compared with those of the other two groups. In the performance-based scenario questions, there was little 
difference in the mean scores of the three groups. Table 1 illustrates the detailed means and standard deviations of 
students’ learning outcomes in posttest I. 

Table 1 Means and Std. Deviations of Posttest I Scores 









Group 






Full Score 






1 


2 


3 


Total 




Quiz 1 


Mean 


36.25 


32.67 


33.89 


34.15 


60 




Std Deviation 


9.59 


11.46 


10.60 


10.64 




Scenario 1 


Mean 


22.55 


22.87 


22.28 


22.55 


25 




Std Deviation 


2.43 


2.33 


2.25 


2.33 





Note: Group 1 — Experimental group with concept map; Group 2 — Comparison group with outline; Group 3 — Control 
group. 



To test hypothesis I, ANOVA was used to compare the mean scores of posttest I of the three groups. Null 
hypothesis I suggests that students who were exposed to a concept map AO or an outline AO would show no 
difference, in both the short-term knowledge-based and performance-based learning achievements, from those who 
were not exposed to an AO. Table 2 shows that there is no statistically significant difference among the three groups 

in either the knowledge quiz (F2, 122= 1.130, a>0.05) or the performance scenario questions (F2, 137 = 0.412, a>0.05). 
Also, the effect sizes for AOs in both tests are relatively low. Only 1.8% of the differences in quiz 1 scores can be 
explained by the treatments of AO among the groups. Fess than 1% of the difference in scenario 1 scores can be 
explained by the use of AO. 

Table 2 Tests of Between-Subject Effects in Posttest I 



Source 


Type III Sum 
of Squares 


df 


Mean Square 


F 


Sig. Partial Eta 

Squared 


Quiz 1 


255.452 


2 


127.726 


1. 130 


.327 .018 


Scenario 1 


63.326 


2 


31.663 


.412 


.664 .009 
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Null Hypothesis II: There is no difference in the long-term knowledge-based and performance-based learning 
achievements among students in the concept map, outline and control groups. 

In posttest II, there are little variations in the mean scores in either quiz 2 or scenario 2 questions. The 
control group outscored the AO treatment groups by less than 1 point in both tests. Table 3 shows the means and 
standard deviations of the students’ learning outcomes in the delayed posttest. 

Table 3 Means and Std. Deviations of Posttest I Scores 









Group 






Full Score 






1 


2 


3 


Total 




Quiz 2 


Mean 


29.04 


30.95 


30.76 


30.36 


60 




Std Deviation 


11.32 


6.86 


8.40 


8.71 




Scenario 2 


Mean 


23.29 


22.23 


24.10 


23.17 


25 




Std Deviation 


3. 62 


5.57 


2.77 


4.21 





Note: Group 1 — Experimental group with concept map; Group 2 — Comparison group with outline; Group 3 — Control 
group. 



Similar to the findings in the short-term learning achievement posttest I, the difference in posttest II is not 
statistically significant in either the knowledge-based quiz (F2,95= 0.412, a>0.05) or the performance-based scenario 



questions (F2,6o= 1.051, a>0.05). The effect size of AO in quiz 2 is below 1%. The effect size of AO in scenario 
questions 2 is 3.4%, far below 20% which is indicative of a small effect by the Cohen’s convention. 

Table 4 Tests of Between-Subject Effects in Posttest II 



Source 


Type III Sum 
of Squares 


df 


Mean Square 


F 


Sig. 


Partial Eta 
Squared 


Quiz 2 


63.326 


2 


31.663 


.412 


.664 


.009 


Scenario 2 


37.13 


2 


18.565 


I.05I 


.356 


.034 



Additional Analyses 

During the study, students were given the opportunity to fill out a survey about their online learning 
experience, especially their attitudes towards using AOs. 130 students from the three groups completed this survey. 
Table 5 shows the students’ experience using AOs in the treatment groups. Approximately half of the respondents in 
the concept map group indicated that they spent 6-10 minutes reading the concept map, and they read it twice. 
Approximately half of the respondents in the outline group reported that they spent 1-5 minutes reading the text 
outline, and they read it once. The majority of the respondents in both groups agreed that using advance organizers 
was helpful for them. 

Table 5 Survey on Students ’ Experience with Using AOs 





Concept Map 




Text Outline 






Mean 


% 


N 


Mean 


% 


N 


Time spent on AO 


6-10 min 


42.1% 


16 


1-5 min 


44.2% 


19 


How many times read AO 


Twice 


57.9% 


22 


Once 


41.9% 


18 


When read 


Before textbook 


44.7% 


17 


Before textbook 


51.2% 


22 


Usefulness 


Agree 


86.8% 


33 


Agree 


69% 


29 


Total 






38 






42 



Discussions 

Neither of the null hypotheses failed to be rejected in this study on using advance organizers (AOs) in 
Web-based distance education. The research findings indicate that there is no difference in either the short-term or 
long-term learning achievements among students in the concept map, outline and control groups. 

According to Ausubel’s assimilation theory, students given AOs should perform better on tests on the 
material-to-be-leamed than students in control groups (Ausubel, 1968). In the current study, students in the 
concept-map group outscored the other groups by 3-4 points out of a full 60 points by average in the immediate 
knowledge quiz (posttest I). The difference in performance-based scenario questions was slight among the three 
groups. The full score for the scenario questions was 25, and the mean scores for all of the groups were around 22.5, 
indicating a ceiling effect that the assessment instrument may lack sensitivity and discrimination in measuring 
learning outcomes. The control group scored at an average of 22.28 out of 25 on the scenario questions. There was 
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less than 3 points (12%) of improvements for the treatment groups to achieve. Similar to the historical studies on 
AOs in face-to-face classes, no statistically significant difference was found in either the knowledge-based or the 
performance-based tests in this Web-based AO study. 

It is speculated that the AO effect should be greater in longer studies, especially in the ones over 10 days, 
than in shorter ones (Ausubel, 1968; Luiten et al., 1980; Stone, 1983). However, this study failed to prove a greater 
AO effect on students’ learning achievements in a delayed posttest (posttest II) four weeks after the AO intervention. 
The differences in both the knowledge-based and the performance-based tests were slight, and the effect sizes were 
considered to be small and below 0.05 by the Cohen’s convention. Similar to posttest I, a lack of differentiation 
might be one of the reasons that attribute to the non-significant result in posttest II. Also, in the posttest, students 
reported a shortage of time during the quiz, and over 50% of the students made errors in the last two quiz questions. 
Therefore, speeding effect might be another factor that seriously affect the measuring error of the quiz instrument. 

In combination with the AO effects, this study examined other factors that might influence students’ 
learning achievements using repeated-measure analysis of covariance (ANCOVA). The results show two 
statistically significant factors: time between posttests and weekly study time. There is a statistically significant time 
effect (FI, 89=13. 185, p<.01) on students’ learning outcome. Almost 13% of the variance in scores can be explained 
by the time elapsed between the two posttests. This indicates that students’ test scores were considerably lower in 
the delayed posttest II compared with the immediate posttest I, with an elapse of four weeks’ time. Also, weekly 
study time has a statistically significant effect on students’ learning achievement (FI, 42=6. 165, p<.05). Almost 13% 
of the variance in scores can be explained by the weekly study time. This suggests that the more time students had 
reported as learning time, the better they would perform in the posttests. Other factors were examined without 
statistical significant effect. They are GPA scores, technical abilities, pretest scores, Web-based learning experience, 
age, and class standing. 

Most of the students found using AOs, especially the concept map, helped them scaffold the learning 
materials. Their feedbacks in the survey indicated how they used AOs in learning. The majority of the students 
would read AOs before they read the textbook. They spent, on average, 6-10 minutes reading the concept map, and 
would usually refer back to the concept map during or after they read the textbook. For the text outline, the student 
would spend 1-5 minutes reading, and read it only once. According to the survey results, this study successfully 
demonstrated how AOs could be integrated in Web-based distance learning, and the concept map was better 
received by the students compared with the text outline. 

Conclusions 

Web-based distance learning is becoming an important trend in the higher educational settings. More and 
more instructors and students choose online classes to take advantage of the time and location convenience. It has 
always been a challenge to examine the effects of pedagogical strategies in a fully Web-based environment. The 
current study investigated the use of advance organizers (AOs) in a fully Web-based health care ethics course. 
Consistent with results of the studies in the traditional classes, this study failed to show a statistically significant 
short-term or long-term effect of AOs on knowledge-based or performance-based learning achievements. Also, this 
study failed to determine which format of AOs (graphic or text) better facilitate students’ learning achievement. 
However, students showed positive attitudes towards using AOs in online learning and they reported that the use of 
AO helped them break down the course contents and highlighted the important concepts. 

The failure to generate a statistical significance might be due to several reasons. One of the issues that the 
researcher had found in the study is that it is impossible for an online quiz to be monitored. Though the quizzes had 
been instructed as close-book tests, it was possible that students still referred to their lecture notes and the textbook 
while they answered the quizzes. This might seriously weaken the validity of the quiz instruments. An important 
implication for further research is to develop measures to prevent students from referring to other assistant materials 
in self-directed online quizzes. Another reason for the non-significant result might be the lack of discrimination of 
the scenario questions in measuring students’ analytical and critical thinking abilities. Future studies need to develop 
more strict rubrics to differentiate students’ learning outcomes in the assessment instruments. Also, the limited 
intervention length might be an important factor that influences the effectiveness of AO. The current AO 
intervention was used in a one-week module. For future studies, it might lead to statistically significant results if the 
AO strategy is integrated in a whole semester-long course and the AO effect is tested on a course level. 

This research shows that instructional strategies, like advance organizers, can be incorporated into online 
learning experience. It has been assumed by many researchers that the adoption of effective online teaching and 
learning strategies is a solution to learning challenges in an interactive multimedia environment. This investigation 
can be served as an example for future studies on how to design, implement, evaluate, and disseminate empirical 
research result on the effectiveness of instructional strategies in Web-based distance education. 
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Abstract 

This study examines four levels of learning outcomes of 204 undergraduates in learning 
environments featured two navigation modes (i.e. linear and nonlinear) and three instruction 
presentation modes (i.e., text, text with static graphics, and text with animation). The results 
indicated that students performed significantly better with the help of visuals in lower-level 
assessments regardless of navigation modes. The implication for instructional design will be 
discussed and presented. 



Introduction 

Designing effective online instructions has gained increasing attention nowadays because of the 
ubiquity of the Internet that leads to the popularity of online learning and distance education. Since most of 
the online instruction and learning materials are delivered through websites, the design principles of 
websites should be considered while designing online learning materials. As suggested by Nielsen (2000), 
typical principles governing website design include navigation, response time, credibility, and content. 
Among these principles, navigation mode is one important feature to be considered especially in designing 
instructional websites because it affects the instructional sequence learners follow and the degree of control 
that learners have over their learning (Lawless & Brown, 1997). To date, a great amount of navigation 
studies were conducted to understand the browsing behaviors of website users; however, only a handful 
studies have investigated its effect on student learning achievement at different levels. 

Taking advantages of the advanced computer technology and the broadband network, web-based 
instruction can now afford to incorporate multimedia designing to enhance learning in terms of the 
accessibility. The use of static visuals and animation in web-based instruction is one example of the 
aforementioned development. As an instructional strategy that provides attractive and explicable visual 
assistance, animation is deemed to be capable of arousing learners’ interests and facilitating learners to 
comprehend the concept (Richer, 1994). In the past two decades, respectable efforts have been devoted to 
studies examining the effectiveness of animation. However, there were no consistent results. Some have 
reported positive effects of animation (Beck & Layne, 1988; ChanLin, 2004; Park & Gittelman, 1992; 
Richer, 1990b; Richer, Boyce, Assad, 1990), while others have revealed no significant differences between 
animation and static visuals (Lin & Dwyer, 2004; Reed, 1985; Richer 1989). One reason to explain the 
inconsistency is that studies measure learning outcomes at different levels from factual knowledge to 
problem-solving skills. It is necessary to differentiate the effectiveness of animation in helping different 
levels of learning. 



Theoretical Framework 



Navigation 

Lawless and Brown (1997) argued that multimedia learning environment allows learners control 
over the information they access and their learning sequences, which helps create meaningful learning. Hsu 
(2006) also indicated that nonlinear capability of the WWW has great potential in education for the similar 
reasons. However, nonlinearity also could lead to disorientation, which poses the potential negative impact 
on learning since some learners might be overloaded with information while lacking a clear ’’roadmap” to 
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proceed with learning (McDonald & Stevenson, 1998). Regardless the disorientation problems, researchers 
(Jacobson & Spiro, 1995) posited that learners with navigational choices did better in transferring their 
knowledge to a new situation. 

Animation 

In educational research, animation generally serves one of the following functions: attention- 
gaining, presentation, and practice (Rieber, 1990a). As for its presentation function, animation can be 
designed to represent domain knowledge and explain complex phenomena (Park & Gittleman, 1992). The 
cognitive theory that guides the instructional use of animation is Paivio’s (1990) dual coding theory. This 
theory suggests two different processing systems in human cognition, one verbal and the other nonverbal. It 
further argues that words and pictures are encoded separately in these two aforementioned systems. 
Accordingly, from a perspective of information processing, animated visuals helps increase the chances for 
information being processed and retrieved. 

In addition to the theoretical basis, like the static visuals, animation should also be placed where 
there is a “need for external visualization” in order for visuals to aid the learning successfully (Rieber, 
1994). Meanwhile, motion and trajectory are deemed to be the attributes possessed by animation that 
distinguish its capability from static graphics (Riber, 1994; Weiss, Knowlton & Morrison, 2002). That is, in 
order for animation to be more effective than static ones, the subject to be learned should involve either the 
motion or the trajectory of an object, or both. Without the need for motion and trajectory, the advantage of 
animation as an instructional strategy will not be shown. 

Research Purpose and Hypothesis 

This study aims to examine two factors, navigation mode and presentation mode of instruction, in 
a web-based multimedia environment. Specifically, there were three purposes of this study. First, the effect 
of navigation mode (i.e., linear and nonlinear) on learning achievement of different levels was investigated. 
It was hypothesized that learners would have better achievement in a nonlinear learning environment 
because learners would have more control over their learning sequences. Second, presentation mode of 
instruction (i.e., text, text with static graphics, and text with animation) on different levels of learning 
achievement was examined. It was hypothesized that learners with the support of animation would learn 
significantly better than learners without the support of animation, hence better performance on all the 
criterion measures. Third, this study intended to investigate if a significant interaction existed between 
navigation modes and presentation modes. 



Methodology 



Participants 

Two hundred and four undergraduate students from an introductory statistical class in a large 
northeastern university were recruited to participate in this study. There were 78 males and 126 females 
among the participants, with the mean age of 19.62 (5Z)=1.59). Participation was volunteered-based and 
additional class credits were given to the students who participated in this study by the instructor. 

Instructional Material and Criterion Measurements 

Developed by Dwyer (1978), the core instructional unit employed in this study contains 
approximately 2000 words about the anatomy and functions of the human heart, including the parts of the 
human heart, circulation of blood flow, and blood pressure. This instructional material comes with five 
criterion measures: comprehension, drawing, identification, terminology, and total. A brief description of 
each criterion measure follows: 

Drawing Test (KR- 20= .93). The objective of the drawing test was to evaluate student ability to 
construct and/or reproduce items in their appropriate context. The drawing test provided the students with a 
numbered list of terms corresponding to the parts of the heart discussed in the instructional presentation. 
The students were required to draw a representative diagram of the heart and place the numbers of the 
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listed parts in their respective positions. For this test, the emphasis was on the correct positioning of the 
verbal symbols with respect to one another and regarding their concrete referents. 

Identification Test (KR- 20= .87). The objective of the identification test was to evaluate student 
ability to identify parts or positions of an object. This multiple-choice test required students to identify the 
numbered parts on a detailed drawing of a heart. Each part of the heart, which had been discussed in the 
presentation, was numbered on a drawing. The objective of this test was to measure the ability of the 
student to use visual cues to discriminate one structure of the heart from another and to associate specific 
parts of the heart with their proper names. 

Terminology Test (KR- 20= .87). This test consisted of items designed to measure knowledge of 
specific facts, terms, and definitions. The objectives measured by this type of test are appropriate to all 
content areas associated with an understanding of the basic elements as a prerequisite to the learning of 
concepts, rules, and principles. 

Comprehension Test (KR- 20= .84). Given the location of certain parts of the heart at a particular 
moment of its functioning, the student was asked to determine the position of other specified parts of the 
heart at the same time. This test required that the students have a thorough understanding of the heart, its 
parts, its internal functioning, and the simultaneous processes occurring during the systolic and diastolic 
phases. The comprehension test was designed to measure a type of understanding in which the individual 
can use the received information to explain some other phenomenon. 

Total Test Score (KR- 20= .96). The items contained in the individual criterion tests were 
combined into a composite test measuring global achievement obtained from the instructional unit. 

Research Design 

A 2X3 posttest experimental design is used to seek answers to the aforementioned research 
questions. Figure 1 presented a graphic illustration of this experimental design. Participants were randomly 
assigned to one of the six treatment groups shown in the figure. 




Figure 1. A posttest-only 2X3 factorial design 



Treatment Material 

Linear navigation. In this treatment, students utilized only “BACK” and “NEXT” buttons at the 
bottom of the text on each frame to navigate. The menu on the left was provided as an anchor without any 
navigation function. Figure 2 is one of the screenshots of the linear-navigation website. 
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4.2. Blood in Auricles 



The Veins and Arteries 
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simultaneously through vein 
openings which have no valves. 
The right auricle receives its blood 
through the superior and inferior 
vena cavas, while the left auricle 
receives its blood through the 
pulmonary veins. 

BACK I NEXT 




Figure 2. Screenshot of a linear navigational frame. 



Nonlinear navigation. In this treatment, instead of the “BACK” and “NEXT” buttons, the 
navigation menu with two levels - the “Unit” level and the “Lesson” level, was provided on the left side of 
the frame (See Figure 3). Lessons embedded in each unit were displayed on-demand using a mouse over 
technique (See Figure 4). While clicking on the drop-down menu displaying lessons, students could move 
directly to any lessons. This learner control was provided consistently on every webpage of this module. 
Thus, students could learn the materials according to the sequence they prefer. 



The 

Human 

Heart 




The Circulation of Blood through the Heart 



The Heart's Structure 





Blood in Auricles 

Both auricles receive blood 
simultaneously through vein 
openings which have no valves. 
The right auricle receives its blood 
through the superior and inferior 
vena cavas, while the left auricle 
receives its blood through the 
pulmonary veins. 



pulmonary 




(►) PLAY ANIMATION 



Figure 1. Screenshot of a nonlinear navigational frame without drop-down menu. 
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Figure 4. Screenshot of a non-linear navigational frame with drop-down menu. 



Text. Students only learn from the text demonstrated on 20 frames with no visual aids. 

Static Graphics. A static graphic was provided on each frame along with the text (the same as 
what provided in the “text” treatment) to facilitate students to visualize the different parts and functions of 
the human heart. Colors and arrows were used for highlighting and for the direction of motion (see Figure 2) 
Animation. In this treatment, in addition to the text, and static graphics, animation was provided 
on 15 frames to enhance students’ understanding of the materials. Given that the function of the human 
heart is viewed as a complex system, animation was used to help learners visualize how the system works 
and improve their conceptual understandings of this system (Weiss et al. 2002). Considering the fidelity of 
animation, following the suggestions of Weiss et al. (2002), abstract animation, instead of concrete one was 
designed to help learners better retain information. 



Various animation techniques were used independently or in combination to provide dynamic 
illustrations of concepts that were difficult to comprehend. Some examples are progressive reveal, motion, 
pop-in verbal, contraction, and expansion. Students were allowed to review the animated visuals as many 
times as needed by clicking on the “PLAY ANIMATION” button (see Figure 3). 



Data Analysis 

Two one-way ANOVA was first used to analyze the data with each of the two independent 
variables, navigation mode, and presentation mode of instruction. The dependent variables were four 
criterion measures and the total of the four tests. Then a two-way ANOVA was conducted to examine the 
interaction effects between two independent variables on the four tests and the composite scores. 



Results 



Results of the ANOVA indicated that the navigation mode did not have a significant main effect 
for all the criterion tests: drawing test F(l, 202) =.999,/>=.319; identification test: F (1,202) = 1.738,/? 
=.189; terminology test: F(l, 202) =.089,/?=.766; comprehensive test: F (1,202) = .264,/? =.608; total 
score: F(l, 202) =.799,/?=. 372. Table 1 presents the means and standard deviations for this factor on each 
criterion test. 



28 





Table 1. Mean and Standard Deviation of Each Criterion Test for Linear and Nonlinear Treatment Groups 

Criterion Measures ^ 



Navigation N 



Drawing Identification Terminology Comprehension 



Total 



Linear 102 9.87 (5.88) 13.34(4.97) 11.88(5.01) 10.14(4.64) 45.24(18.48) 

Nonlinear 102 10.75(6.57) 14.26(5.02) 12.09(4.84) 10.45(4.06) 47.55(18.48) 

^ The maximum score for each criterion measure is 20. 

^ Value in parentheses indicated standard deviation. 



Results of the ANOVA indicated that presentation mode of instruction has a significant main 
effect on students’ scores for drawing test (F (2, 201) =24.064, /?=.000), identification test (F(2, 201) 

=1 1.458,/7=.000) and total score (F(2, 201) =10.305, /?=.000). The effect of presentation mode was 
approaching significance on students’ scores for terminology test (F(2, 201) =2.827, /?=.062). However, 
this main effect was not found significant on students’ scores for the comprehension test (F(2, 201) =1.694, 
/7=.186). Table 2 summarized the descriptive statistics of this main effect on scores for each criterion test. 
Since the omnibus tests are significant in drawing test, identification test and total score, Scheffe’s test has 
been used for the post hoc comparisons. The results of the post hoc comparisons showed that students who 
received either animation or static visuals performed significantly better than did those who only received 
text for the drawing test, identification test and total score. However, animation did not have a significantly 
better effect than did static visuals in all three previously mentioned tests. 



Table 2. Mean and Standard Deviation of Each Criterion Test for Text, Static Visuals, and Animation 

Groups 

Criterion Measures ^ 



PKG 


N ■ 


Drawing 


Identification 


Terminology 


Comprehension 


Total 


Text 


68 


6.47(4.85) 


11.60(5.07) 


10.93(4.65) 


9.68(4.58) 


38.68(17.16) 


Static 


68 


11.82(6.10)’’ 


14.47(4.78) 


12.13(4.92) 


10.18(3.74) 


48.60(17.72) 


Visuals ^ 
Animation 


68 


12.63(5.86) 


15.34(4.42) 


12.90(5.04) 


11.03(4.63) 


51.90(18.15) 



^ The maximum score for each criterion measure is 20. 

^ Value in parentheses indicated standard deviation. 

both Static Visuals and Animation Groups also receive the same text as that received in the Text Groups 



A 2X3 ANOVA was conducted to examine the interaction effects between navigation mode and 
presentation mode. No significant interaction was found between navigation mode and presentation mode 
for all four criterion tests and the total score. Table 3 provided the means and standard deviation of each 
criterion test for all the treatment groups, (drawing test F (2, 198)=.515, p=.598; identification test, F (2,198) 
=.029, p=.972; terminology test, F(2, 198)=.007, p=.993; comprehension test, F (2, 198) =.529, p=.590; 
total score, F (2, 198)=.161, p=.851). 



Table 3. The Relationships of Navigation Mode and Presentation Mode on the Criterion Measures 







Linear 






Nonlinear 






Text" 


Static 

Visuals 


Animation 


Text 


Static 

Visuals 


Animation 


Drawing 


6.26 (6.05) ^ 


10.82 


12.53(4.47) 


6.68 


12.82(6.40) 


12.74(5.60) 


Identification 


11.03(4.69) 


14.06 


14.94(3.52) 


12.18 


14.88(4.45) 


15.74(3.84) 


Terminology 


10.88(4.97) 


12.00 


12.76(3.52) 


10.97 


12.26(5.08) 


13.03(4.11) 


Comprehension 


9.65(5.41) 


9.59 


11.18(3.65) 


9.71 


10.76(3.24) 


10.88(3.68) 


Total Score 


37.82(18.64) 


46.47 


51.41(13.97) 


39.53 


50.74(17.51) 


52.38(15.44) 



^ N=34 for each cells. 

^ Value in parentheses indicated standard deviation. 



29 










Discussion and Conclusion 



The results of this study showed that navigation mode did not have a significant effect on student 
learning achievement across different levels of learning. Students receiving nonlinear instruction performed 
similarly to other students assigned to the linear treatment. A possible explanation is that learners in the 
nonlinear environment might have interacted with the learning materials in a linear fashion by following 
the same order imposed by the drop-down menu, indicating that they could not take the advantage of 
learner control. It should be noted that the drop-down menu provided in the nonlinear environment was 
designed to allow students to access a certain topic by just one click. This design should empower learners 
to explore the materials in the order they prefer, and to revisit certain materials with great convenience as 
they see necessary. However, since the drop-down menu was organized with the sequence of the learning 
materials, should learners not take the advantage to navigate with the sequence of their own choice, they 
would end up reading the materials in the same order as would those students in the linear environment. 
This explanation echoed the previous research findings that, even in a nonlinear environment, learners 
generally relied on the structure of the navigation choices to guide their navigation behavior (Lawless & 
Brown, 1997). For future studies investigating how navigation behaviors affect learning, we would suggest 
that the navigation menu be designed in a way not implying the sequence of learning materials so learners 
are more likely to navigate at their own preferred sequence. 

Presentation mode of the instruction (ie., text, text with static graphics and text with animation) 
had effects on the two tests assessing lower-level achievement, but did not have effects on the two higher- 
level ones. Further examination of this effect on the two lower-order tests showed that learners with the 
aids of both animation and static visuals outperformed learners learning only from text on two lower-level 
tests, with no difference between animation and static visuals. In short, students in animation group did not 
outperform those in the text group in the higher- order learning, neither did it outperform static graphics in 
all four criterion measures. Why did animation fail to help learners learn a complex system involving 
movements? According to the Apprehension Principle (Tversky, Morrison & Betrancourt, 2002), learners 
might have problems in perceiving and comprehending the provided animation. That is, given that the 
learning material is about the complex system of human heart, even though the animation had presented the 
simultaneous movements of different parts in the heart, learners might not be able to perceive the dynamics 
comprehensively and correctly and further comprehend how the complex system works due to the 
limitation of human cognitive capacity. Following this line of thinking, we would suggest that learners first 
perceive each step in the movement correctly then they are at the better position in comprehending the 
entire system. Static visuals portraying movement chunked into discrete steps may help learners perceive 
the subcomponents and how they work correct (Tversky et al., 2002). Then, the dynamics in the systems 
could be presented to the learners using animation. Viewing the animation at this time could help learners 
link the discrete steps together and test their understanding of how the system works. 

Limitations 

This study did not record the learners’ navigation behaviors for analysis. To better understand the 
sequence that learners proceed with the learning materials and analyzing its impact on their learning 
achievement, the navigation behaviors are suggested be recorded and examined in the future study. 
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Abstract 

This study tells the story of the Central Educational Center (CEC), a charter technical high school in 
Newnan, GA, that represents an attempt by Joe Harless to apply his industry validated performance technology 
ABCD system to some of the problems facing public education. Now in its 7^^ year of operation, the school has 
received both local and national recognition as a model for educational reform both in terms of technical education 
and a place where the application of educational best practices in general is taking place. CEC, through its 
performance-based technically oriented, yet demanding courses, has been linked to significant drops in county-wide 
drop-out rates (as high as a 60% reduction), increases in academic test scores (graduation tests and SAT scores), and 
significant positive economic impact to the local community (over $75 million). Major stakeholders report high 
levels of satisfaction on a longitudinal basis although CEC faculty report some difficulty in adjusting to their 
changing roles as CEC responds to the evolving course demands of students and industry. 27 CEC students, 
stratified across three academic performance groups (high, medium, and low) were followed from their 10^^ grade to 
12^^ grade years and compared to a control group of students across demographic variables, GPA, and instances of 
unexcused absences and tardiness. The data showed no improvement in academic performance as measured by GPA 
for either the CEC or control group but an independent samples t-test showed a statistically significant effect for 
instances of tardiness, ^(52) = -2.71 ,p< .001, with the CEC group having a substantially lower number of instances 
of tardiness than the control group at the 12^^ grade level. In addition there was a moderately negative relationship 
between performance level and instances of unexcused tardiness at the 12^^ grade level, r(25)= -.44, p<.01, 
suggesting that low and medium performing students decreased instances of unexcused tardiness from 10^^ to 12^^ 
grade while their counterparts in the control did not. Both qualitative and quantitative methods were used to conduct 
this study and the results strongly suggest a meaningful role for the application of system’s principles in public 
education exists. 



Introduction 

''If it is arrogant to offer different paradigms for what and how our educational system teaches our young, I can 
only state my reasons for such immodest behavior. I have spent my entire professional life researching, developing, 
and teaching the methodologies for influencing people - especially in the domain of influencing their knowledge and 
skills'' (Joe Harless, 1998, p. 4) 

This is a story worth telling. Imagine if you took one of the world’s foremost experts in human 
performance, provided him with generous resources along with strong community and government support, and 
asked him to build a utopian, model school from the ground up. The human performance technologist is Joe 
Harless, considered the founder of front end analysis (FEA) and one of the forefather’s of the field itself, and the 
school opened its doors in 2000 in a small Georgia community with tremendous fanfare and promise. Now six years 
later and, after gamering a prestigious National Model High School in 2004 from the Bill and Melinda Gates 
Foundation and being deemed the Georgia model for statewide educational reform, the impact and results of what 
Harless calls his “grand experiment” are examined in this study drawing from five years of research data. 

Upon retiring after 30 years as a human performance technologist. Dr. Harless finally was able to do what 
he had always wanted to do when he first started his career - use performance technology to help improve public 
education. Some of the highlights of Dr. Harless’s career included helping eradicate smallpox worldwide with the 
US Center for Disease Control, preventing additional nuclear meltdowns after the Three Mile Island incident, the 
design and development of the Boeing 111 jetliner, designing the training systems for the US Coast Guard, and 
much of the design and development system used by Microsoft today. In his book The Eden Conspiracy: Educating 
for Accomplished Citizenship (1998), Harless painted a vivid utopian educational vision utilizing his performance 
technology model called the Accomplishment Based Curriculum Development system or ABCD. This grand vision 
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saw an active “conspiracy joined by the entire community to prepare the young for accomplished citizenship” 
(Harless, 1998, pg. 24). In this conspiracy, the desired educational goals would reflect tangible student 
accomplishments where, outside of coverage of a specific set of subject matter-based curriculum or general 
preparation for future work life, students would be prepared to enter society as active, contributing members as 
envisioned by the local community. This school would, furthermore, represent a “grand experiment” for how the 
application of system’s principles would look when brought to bear in public education. 

Harless was on the last chapter of his book in 1996 when the Coweta County Superintendent of Schools 
approached him about a fundamental problem, echoed similarly by districts across the country, with the local school 
system: local businesses and employers were complaining that high school graduates were not meeting minimum 
employment needs. Dr. Harless told the Superintendent that at the heart of human performance technology is a 
customer-driven focus where what the customer should be able to do is first clearly identified and then, by working 
backwards, the appropriate performance system is built around helping customers achieve the desired behavior as 
efficiently and effectively as possible. He told the superintendent that the first step in the process would be to ensure 
that the effort to address issues around employable skills of high school graduates must be those that are having the 
problem and need to identify what skills are required in the first place: businesses. 

Following the principles of his performance technology model, the first step in the design of this ideal 
school was a front-end analysis, in this case a needs assessment of local businesses that identified the tangible 
accomplishments employers expected from newly hired employees. These accomplishments then served as the 
focal point for building a public charter school that was closely aligned with a curriculum designed specifically to 
provide students with the knowledge, behaviors, and skills necessary to achieve these accomplishments. 

Understanding the Context of Nationwide Educational Reform 

What is occurring at CEC represents a local attempt at improving the quality of education but takes place in 
a much larger national context. Public outcries for the need to improve public education have been a prominent, 
consistent aspect since first implemented in the United States during the mid- 1800’s (Ravitch, 2000). Successful 
reform efforts, however, have been few and far between. Contemporary educational reform efforts still largely 
derive their roots from the 1983 National Commission on Excellence in Education’s report^ Nation at Risk: the 
Imperative for Education Reform, which called for a national educational reform movement to meet our “tide of 
mediocrity.” During the original commission’s comprehensive 18 month study involving the nation’s public and 
private schools, the commission found that U.S. student performance was alarmingly low across a number of 
different measures. The report found approximately 23 million American adults were functionally illiterate and that 
among America’s 17 year olds 13% were functionally illiterate while for minority youth this rate was as high as 
40%. SAT scores had also steadily declined (by 50 points verbal, 40 points math) since 1967. Business and military 
leaders were complaining that America’s youth did not have the basic skills expected costing them millions of 
dollars in remedial programs {A Nation at Risk, 1983). The results were even more disturbing given the report’s 
assertion that the U.S. was not adequately preparing its nation’s youth for the highly skilled labor force needed to 
compete with other industrialized nations {A Nation at Risk, 1983). Finally, in an international comparison of 19 
tests with other industrialized nations, U.S. students were last seven times. 

Although some researchers have claimed that the perceived problems in education identified in the mid- 
1980’s were in many ways a myth manufactured largely by the conservative Reagan and Bush administrations 
(Berliner & Biddle, 1996; Bracey, 1996), it is clear that the primary areas of disagreement lie with how to interpret 
the general student performance measures and how we compare internationally. What both sides do agree on, 
however, is that there is a real decline in American education and, although at times over exaggerated especially 
during international comparisons through the powerful constituencies and entrenched bureaucracies within public 
education that do not favor major changes, there are indeed legitimate problems with schools where too many 
students are performing poorly and that many discussions on the failure of schools serve as proxies for much more 
complex larger societal issues that are largely beyond the scope of educational by itself (Berliner & Biddle, 1996; 
Stedman, 1996; Waters, 1998). In a 1996 report by the Sandia National Lab, it was concluded that viewing 
American education strictly as an aggregate measure is not precise enough and that the “fine structure” of the 
nation’s aggregate data pointed to significant dropout problems among minority youth and students in urban schools 
(70% of black students and 50% of Hispanic students graduated on time compared to 80% of white students) and an 
in lower minority standardized test scores than whites students (black students averaged nearly 20 points lower 
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while Hispanic and Native Americans scores were lower by more than 100 points). Interestingly enough, a review of 
workforce data showed “very few companies point to inadequate academic preparation for new employees but rather 
focus on social ’’skills” such as punctuality and personal appearance” (Husselkamp, Carson, and Woodall, 2000, pg. 
46) 



In 1998, a panel of national educational experts came together to examine the progress the nation had made 
15 years after the Nation At Risk report had been written. What the attendees found was that little or no academic 
progress has been made in the 23 years after the 1983 A Nation at Risk Since 1983, over 10 million students 

had managed to reach 12th grade still functionally illiterate while over 6 million students had dropped out entirely. 
For minority groups the results were even worse - in 1996 13% of black youth aged 16-24 were not in school and 
did not hold a high school diploma; 17% of first generation Hispanic youth had dropped out of high school and 44% 
of all Hispanic immigrant youth also dropped out of high school (Fordham Foundation, 1999). Their findings 
entitled A Nation Still at Risk concluded that “a dual school system had been created” where the gap between good 
and bad schools had widened. Educational access to high quality education was labeled the next civil rights 
movement {A Nation Still at Risk, 1999). The report stated, “The United States is the only country in the world in 
which children fall further behind the longer they stay in school” {A Nation Still at Risk, 1999, pg. 3). The report 
recommended that schools focus on three primary factors: standards, assessment, and accountability with a specific 
emphasis on pluralism, competition, and school choice. 

A Call for a More Holistic, Systems Oriented Approach 

How come so much effort and so little return on investment? There has been a rallying cry by many that the 
multitudes of “waves” of reform that have been tried over the past two decades represent “piecemeal” change and 
meaningful, widespread reform cannot occur unless systemic changes are made (Meier, 1998; Reigeluth, 1993; 

Betts, 1991; Hawley, et al, 1988). Harless’s The Eden Conspiracy (1998) and the subsequent opening of the Central 
Educational Center based on its tenets, represents the systemic approach called for by many educational reformers 
and utilizes the existing flexibility and autonomy presented by a contemporary, extremely popular educational 
reform initiative - charter schools. In addition, Harless’s reform effort represents a unique opportunity to observe the 
impact of system’s principles using the entire ADDIE system’s process (Analysis, Design, Development, 
Implementation, and Evaluation) from start to finish on a public school and its impact on overall student 
achievement. One of the central problems with education Harless reasons is not HOW we are teaching but rather 
WHAT we are teaching. He suggests that we develop schools and curricula based on desired accomplishments 
students should be able to perform rather than a set of separate content areas deemed to be intellectually appropriate 
for an educated, democratic population. He emphasized the need to make education relevant for students. 

Research Questions and Rationale 

The rationale for the need to study and understand CEC is strong. This study attempts to determine the 
overall integrity in which the CEC implementation follows system’s principles and what the impact the use of 
system’s principles and human performance technology has had within the context of the school, school district, and 
overall student performance. In addition, if CEC is found to be a valid, successful instance of the application of 
system’s principles in public education, how would we go about replicating it successfully outside of Newnan, GA 
and without someone of Joe Harless’s experience to help design and implement it? Specifically, this study has 
explored and sought to answer two major research questions: What have been the results of using a system ’s 
approach to design, develop, and implement the Central Educational Center? And, what impact has CEC had on 
student performance during its initial five years of operation? 

Research question two looks specifically at student performance during CEC’s first five years. This 
question looks closely at how student performance is measured at CEC within the larger context of school 
performance. What are the overall goals of CEC, what indicators have been identified to measure these goals, and 
what instruments and data are collected to measure these indicators? The accurate measure of performance is 
always of critical importance and, as CEC continues to gamer state and nationwide acclaim, the need to ensure that 
the indicators of success being used by the school are accurate, valid, and reliable have become increasingly 
important. Most critically is to ensure that the existing performance and anecdotal data that have earned CEC much 
acclaim and national recognition is clearly linked to overall student performance measures. 



34 




Method 



In order to protect the overall validity and reliability of the study’s findings a mixed-method design was 
used emphasizing the case study supplemented by survey research and historical, longitudinal data. The case study 
was selected as the most appropriate method for studying a uniquely complex, social system such as the Central 
Educational Center. Stakeholder survey research, defined as the solicitation of “self-reported verbal information 
from people about themselves” (Rea & Parker, 1997, pg. 2), was also used to establish past and current school 
climate and stakeholder satisfaction that helped inform the overall research foundation used for exploratory and 
explanatory purposes. Online survey instruments, focus groups, and a review of historical data with a specific 
emphasis on longitudinal school and student level performance data were also part of the data collection process. 

Case and Units of Analysis 

To implement a systematic approach in conducting and analyzing the results of a case study of CEC, the school 
has been examined through the use of multiple embedded units of analysis utilizing two different models: 
Stufflebeam’s (2002) Context, Inputs, Process, and Product (CIPP) evaluation model and Gharajedaghi’s system’s 
model of function, structure, and process. The primary case is the Central Educational Center. Within CEC, 
however, the perspectives of the primary stakeholder groups (students, faculty and staff, administration, businesses, 
and parents), have been collected and examined. In addition, historical performance data has been gathered and 
analyzed against the school’s stated charter goals and objectives over a five year longitudinal period. 

School performance has been assessed using a combination of criterion referenced outcomes identified in its 
original charter supplemented by stakeholder qualitative comments. Student performance has been examined by a 
number of methods including the longitudinal value-added impact CEC has had on students compared to their non- 
CEC peers and the general district level performance measures associated with the school’s charter. 

Participants and Sampling 

Students: Twenty student interviews, two focus groups, and five class room observations were conducted 
using a combination of purposeful stratified sampling and samples of convenience (representatives selected by the 
school’s guidance counselor). The purposeful sample was one male and one female from each grade level 9-12. In 
addition five natural observations took place each focusing on a different type of course offered at CEC - An 
academic course (a math course), a technical high school course (pre-engineering), a technical college course (dental 
assisting taught by the technical college), one work-based learning course, and the performance learning center 
which is a virtual high school credit recovery program. 

To determine the value-added impact attending CEC has on student performance longitudinally, two 
groups of students were selected - an intervention and control group. For the intervention group (n=27), students 
were selected based on the criteria that they had attended CEC for at least two semesters by the end of their 
sophomore year and were members of one of three pre CEC performance groups based on GPA: High (88%+ or 
3.5+ GPA), Medium (70-88% GPA or 2.8 to 3.5 GPA), Low (<70% or <2.8 GPA). The entire population of 
students fitting the CEC student criteria was 32 10^^ grade students taken from the 2002-2003 academic year and 
followed through the 2004-2005 academic year, the year of their expected graduation. This CEC cohort did not 
distribute equally across the stratified pre CEC performance categories with High (n=10, 31.3%), Medium (n=19, 
59.4%), and Low (n=3, 9.4%) and five participants had to be dropped because they did not remain in the school 
system for the entire three year period. The control group of 27 students represented a stratified sample designed to 
match the CEC intervention group in terms of performance stratification, gender, and ethnicity. The only variable 
not matched between the treatment and control groups was base high school. 

Data from two 2004 surveys, a satisfaction survey with a sample size of n=490 and a survey conducted by 
ICLE, International Center for Leadership in Education, with a sample size of n=300 was also used. 

Parents: Four parents were also interviewed, three of which were samples of convenience arranged with 
the school counselor and one selected through purposeful sampling, a parent of one of the students selected through 
purposeful sampling. Data from a ICLE 2004 survey with a sample size of n=64 was also used. 
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Faculty and Staff: Five CEC faculty and six CEC staff members were interviewed using a combination of 
random selection and purposeful sampling from the CEC directory. Four faculty and one staff member was selected 
randomly from an alphabetical list where their names were selected at random from the list until the desired number 
of interviewees were identified. The randomly selected staff member was chosen by picking one staff member from 
the CEC directory. 

Two additional surveys were administered to CEC’s faculty and staff: the Quality School Assessment 
Instrument (QSAI) and a faculty and staff satisfaction survey. Overall response rate was 41% (n=27) for the QSAI 
and 45% (n=30) for the faculty/staff satisfaction survey. Both surveys were sent out via faculty/staff email list serve. 

Educational Administrators: The current and past administration for CEC and West Central Technical 
Center were interviewed - five current and former CEC administrators and two current and former West Central 
Technical Center administrators. In addition, each of the three principals of the base high schools, which provide 
CEC with its students, were also interviewed at their respective schools and selected because of their unique 
perspectives and leadership roles of the schools that partner with CEC on a day-to-day basis. The Superintendent 
was also interviewed and shared his opinion of how CEC has impacted the school district and his vision for its 
future. 



Business: Four business interviews were also conducted based on a combination of convenience sampling 
through the assistance of school’s CEO and purposeful sampling based on a business’s ability to answer specific 
questions. One business was a work-based learning partner, another business was a temporary employment agency, 
and the final two were original players with CEC before the school opened. 

Community Leaders: Joe Harless was interviewed at length as the originator of the CEC concept. In 
addition, the original business leader that initially approached the Superintendent about problems with overall 
student quality was also interviewed at length. 



Results 

What is the Central Educational Center? A charter school with a twist. 

The Central Educational Center first opened its doors in August 2000 and just completed its sixth academic 
year. It is a non-immersion, charter technical high school that draws students from three base high schools with an 
average district attendance of 4,746 students per year. Students who attend CEC must work through their school 
counselors to identify and select courses at CEC that are not offered at their traditional base high schools. Students 
do not attend CEC on a full-time basis and earn their high school diplomas from their base high school. Another 
unique facet of CEC is that a Georgia technical college, the West Central Technical College, is also located on their 
campus so that a select group of qualified high school students are able to take dual-enrolled courses. Two key 
differences between CEC and most other charter schools are 1) rather than serving as an accountability measure that 
competes with traditional high schools for students, CEC shares students with their respective base high schools 
framing the relationship in a collaborative rather than a competitive manner and 2) CEC has partnered with a 
technical college sharing both facilities and faculty thereby enriching its instructional capabilities above and beyond 
support provided by the local school district and state department of education. 

In terms of general demographics, 615 students attended CEC during its inaugural year (15% of the district’s 
student population) and general student population at the school has increased to an average attendance of 1304 
unique students per year or 27% of the district’s student population. Ethnically, the CEC student population over 
the past four academic years on average has been 73% white, 23% black, 2% Hispanic, and 2% Asian/other. In 
terms of gender, CEC is on average 60% male and 40% female. In terms of grade level, CEC is on average 
comprised of 6% freshmen (n=67), 28% sophomores (n=320), 39% juniors (n=440), and 27% seniors (n=310). 
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CEC Grade Breakdow n - 2001 to 2003 Academic Years 
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Figure 1 - CEC breakdown by grade 



The school district has centralized the majority of technically and career oriented courses at CEC and, although 
there are a few traditional academic courses offered as well, the majority of students (on average 88% or n=1000) 
pursue high school technical/career seals of endorsement, which are earned by taking four selected courses in a 
particular program of study. A small number of students, on average 12% or 138 CEC students per year, take and 
pass the state’s technical college entrance exam as a requirement to getting accepted into the dual enrollment 
program allowing them the opportunity to take dual-enrolled courses at the technical college. Adult learners, on 
average 35 per year, also take courses side-by-side with high school students. 

Research Questions 

Research Question 1: What have been the results of using a system ’s approach to design, develop, and implement 
the Central Educational Center? 

The proposition that needs to be first addressed in order to answer this research question is to determine the 
validity of the following statement: There is a high level of congruence between what is actually in place and 
Harless ’s ABCD system model. The pattern that would be expected if indeed this proposition was true is adherence 
to ABCD system’s process are clear goals that have been identified, resources are identified and allocated 
accordingly, stakeholder satisfaction levels are high, and frequent evaluation resulting in continuous improvement is 
occurring. An invalid pattern would be that no clear objectives are identified, poorly aligned resources, low 
stakeholder satisfaction levels, and either infrequent or no evaluation is taking place. 

The results of the study suggest that CEC matches more closely with the valid pattern, which be predicted 
since Dr. Harless himself led the initial design and planning process. CEC was designed carefully over a four year 
period from 1997 to 2000. One of the essential elements attributed to the success of CEC is the pervasive 
commitment begun by the original steering committee of involving all of the major stakeholders in the decision 
making process from the very beginning. A number of the original steering committee members interviewed 
remarked how they were skeptical that the steering committee would be productive and successful, especially given 
so many disparate points of view and potential “turf battles.” In a comprehensive interview with Dr. Harless (May 
2006), he rated his overall satisfaction level with CEC a 5 out of 10. He was quick to point out, however, that this 
was a criterion referenced rating based on his high expectations and utmost fidelity to his accomplishment based 
systems model (the ABCD system). If norm referenced, compared to other schools, the rating would be much 
higher, especially since CEC has indeed become a state and national model for educational reform. CEC to him is 
the first pilot test of the vision articulated in the Eden Conspiracy. 
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Dr. Harless noted that in recent years he has added an “A” in front of the traditional ADDIE systems design 
process. He affectionately calls the new model “Aunt” ADDIE with the new “A” standing for assessment of 
customer needs prior to an analysis of performance problems. The new model is A- ADDIE. There were a number 
of goals the he had wanted to see achieved in the implementation of CEC. The first was to demonstrate that a new 
“A” needed to be added to the ADDIE process. If done successfully, the resulting school would be a first attempt at 
implementing what he calls needs-based and customer driven education as opposed to the traditional educationally 
mandated, content based, college focused system we have now. He also saw CEC as a demonstration project of 
what could happen if content was derived directly from desired ends-based performance or what he calls 
accomplishments, ends-based quality outcomes. He also wanted CEC to demonstrate the full power and effect of 
implementing the instructional science called for by instructional systems design principles. 

Although, Dr. Harless overall, felt that much of this has indeed been accomplished to some extent, his 
disappointment rests at a number of key areas along the process. First, the use of the needs assessment, the actual 
implementation of the results was not done well. He felt that the initial needs assessment represented a macro needs 
assessment and that what needed to follow was a micro needs assessment to flesh out the specific behaviors and 
knowledge and skills required by employers in these identified areas. In terms of the use of instructional science, he 
felt that he was not provided enough time to train CEC’s faculty (he was given 35 hours or one work week to 
provide training) in the complex application of a science he spent over 30 years to perfect. Because of this, he felt 
that he overwhelmed most of the faculty, and that because of this little or no instructional science is taking place in 
the classroom at CEC. The third issue rests with a disconnect between true performance-based instruction and 
content, standards-based instruction. Performance or accomplishment-based instruction works backward to link 
accomplishment to behaviors that then informs what knowledge and skills need to be taught. Content-based 
instruction although largely driven by behavioral objectives is missing the direct link to desired accomplishments as 
defined by the needs of the customers who will be hiring students. One last concern was that although there was 
considerable evaluation conducted by the author and other research organizations, there had not been any systematic 
follow-up with graduates or the business stakeholders to determine satisfaction levels and/or needed redesign of the 
curriculum. 

Primary Stakeholders Report High Levels of Satisfaction 

CEC students have consistently reported high satisfaction ratings over the past five years. During the data 
collection that took place May 2006, two student focus groups and over 20 individual interviews of 32 students 
yielded an average satisfaction rating of 93%. Students like the different classes they can take at CEC that are not 
offered at their base high schools and also feel like they are treated more like adults in a college campus atmosphere. 
The strength of the teachers and the preparation for the work force through hands-on activities also were mentioned 
as tremendous positives. In addition, although usually mentioned as a weakness, the fact that CEC’s attendance and 
tardy policies are so strict helps “make people come to school.” The fact that students could earn dual enrolled high 
school and technical college credit completely for free also was mentioned as a strong aspect of CEC. 

Parents also report high levels of satisfaction. An independent survey of CEC parents conducted by the 
International Center for Leadership in Education in 2004 found parents to be either agree or strongly agree to 
statements that CEC possesses recognized criteria for effective schools such as safety (95%), my child knows what 
is expected of him/her (97%), and the school is helping develop my child (94%). All four of the parents interviewed 
during the most recent data collection reported general satisfaction of 9.5 out of 10. The only rating not a 10 was an 
8 due to some large class sizes in some of the academic courses (i.e. economics) and also because CEC is bit sparse 
in its technology curriculum. Other factors behind the ratings include “CEC has all of the advantages” and that their 
children really like CEC. Per one parent, “Experience is usually the best teacher; (CEC helps) channel their energy 
to help themselves.” 

Faculty, staff, and administration also report high levels of satisfaction. A faculty and staff satisfaction 
survey has been conducted at CEC every other year, a total of three times over the past six years. The satisfaction 
ratings have been consistently high with a 9.3 out of 10 rating (n=25) in 2002, 9.3 out of 10 in 2004 (n=37), but fell 
to an 8.0 in 2006 (n=30). Another faculty survey conducted by the International Center for Leadership in Education 
(ICLE) in the spring of 2004 as part of the Bill and Melinda Gates’ National Model High School Award. 33 faculty 
and staff responded to the survey. The faculty overwhelming agreed that the school had high academic standards 
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(97% agreed, n=10, or strongly agreed, n=22), had a clear mission (100% agreed, n=6, or strongly agreed, n=27), 
and set high expectations for quality teaching (97% agreed, n=6, or strongly agreed, n=26). 

In terms of general satisfaction level, four directors were interviewed in May 2006. One director called his 
job at CEC a “teacher’s dream” with students who want to be there, small classes, long lunch breaks, and plenty of 
planning time along with no unnecessary meetings or activities and plenty of opportunity to experiment. Another 
director noted that she had taught at different high schools and they were much more “massive.” At CEC there is 
more of an adult atmosphere and students rise to the occasion and behave accordingly. In addition the administrative 
support is outstanding. One director noted that he was happy with CEC but felt frustration over not growing fast 
enough as there is so much he wants to do personally and professionally. Another director noted there “is always 
room for improvement” and that in many ways it was the negative attitudes of the base high schools that were a 
product of growing pains that caused a certain degree of dissatisfaction. In terms of their colleagues, however, they 
felt that the satisfaction was a lot lower at 63%. There was frequent mention of educational bureaucracy and the 
high level of turnover of CEC administration (all three CEC administrators left at the end of the 3^^ year). One 
director felt that there was a high level of satisfaction but that some bad habits had formed during the time that all 
three of the founding administrators left after year three. Another prominent source of dissatisfaction is with general 
communication with existing administration and a feeling that they are simply “treading water” and “missing 
direction” but they cannot quite put their finger on what the problems are as a faculty. 

The faculty also recently completed a school climate survey called the Quality School Assessment 
Instrument (QSAI) which seeks to measure how a school compares to factors identified in the school effectiveness 
literature. 27 faculty, staff, and administrators completed this instrument online. Some results of interest include the 
fact that the top rated item Question 1 1, “Attractive, safe facilities with adequate physical resources” was the only 
item that had an average rating above a 4.0, which is slightly above “at expectation”. The second highest rated item 
was Question 42, a report card item, which asked “how safe is your school?” A 3.45 rating suggests that CEC 
faculty and staff feel safe in their school environment. The QSAI report card, the final 9 questions, “grades” a school 
on nine essential factors of effective schools. On these nine items the grand mean was a 3.2 out of 5, which 
translates to a slightly above average rating. Items of interest include responses to Question 50, “How do you grade 
your school overall?” (3.1 rating). Question 45, “What is the quality of teaching at your school?” (3.3 rating), and 
Question 48, “Does your school provide a quality experience for students?” (3.4 rating). 

The Superintendent of the Coweta County School District and the principals of the three base high schools 
also expressed high levels of satisfaction with CEC. Each of them felt that CEC was good for Coweta County and 
did indeed do good things for their students that attended. Positives noted include strong programs and technology, 
teaches students responsibility, and allows students to see a broader spectrum of the work world. The Superintendent 
also sees CEC and charter schools as representing the flexibility to “think out-of-the box” and try new things 
without being penalized. The shared concerns of the principals can be categorized into two primary problem areas: 

1) Serving the largest number of students who most benefited from vocational education and 2) Administrative and 
operational problems caused by their students attending CEC at various hours of the day. The first issue has to deal 
with total numbers of students served through technically oriented courses. Two of the three principals explicitly 
noted that CEC was not serving enough students. 

CEC School Performance 

An analysis of the school’s performance across its charter objectives indicates that the school has had 
success achieving these stated objectives. Of three measurable charter objectives, CEC achieved its county drop out 
rate target of less than 8% each year (3.10%, 5%, 5.12%, and 4.93%, respectively). In addition, in terms of first time 
pass rates on the Georgia High School Graduation Test, of the four subject areas CEC directly impacts (writing, 
English/Language arts. Mathematics, and Social Studies), CEC achieved its county wide target 100% of the time in 
Science (4 of 4 times), 75% of the time (3 of 4 years) in English/Language Arts, 50% in Mathematics (2 of 4 times), 
and 0% in Writing (0 of 4 times). 

One of the primary problems, however, of trying to use county wide aggregate data is that attempting to 
establish any kind of causality to CEC is speculative at best. During each of its first six years, on average 25% of all 
Coweta County high school students have taken at least one course from CEC each academic year. In order to more 
clearly determine the impact of CEC on student performance two groups of students were tracked over a three year 
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period - a treatment group of CEC students (attended at least two semesters by the end of the 10^^ grade year) with a 
sample size of n=32 and an identical control group of students who did not attend CEC at all also with a sample size 
of n=32 - stratified across three performance groups (high, medium, and low) based on GPA prior to their 10^^ grade 
years. 



Comparing the two groups across repeated measures over a three year period on three variables, GPA and 
unexcused absences and instances of tardiness yielded interesting results. Overall average GPA over a four year 
period from 9^^ grade to 12^^ grade little overall change with either the CEC Treatment group (increased 0.83 points 
from 83.13 GPA in 9^^ grade to 83.96 average GPA in 12^^ grade) or the Control group (increased 0.03 points from 
82.67 GPA in 9*^ grade to 82.19 average GPA in 12*^ grade). Trends in unexcused absences and instances of 
tardiness, however, indicate clear differences between the two groups. For unexcused absences the CEC treatment 
group dropped an average of 25% from 10*^ to 12*^ grade compared to only 6% for the non-CEC control group. See 
Figure 2 below. 

For unexcused instances of tardiness the difference was even more dramatic with the CEC treatment group 
reducing instances of tardiness by a statistically significant 56% from 10*^ to 12*^ grade while the non-CEC group 
actually had an overall average increase of 51%. See Figure 3 below. An independent samples t-test showed a 
statistically significant effect for instances of tardiness, ^(52) = -2.71 ,p< .001, with the CEC group having a 
substantially lower number of instances of tardiness than the control group at the 12*^ grade level. In addition there 
was a moderately negative relationship between performance level and instances of unexcused tardiness at the 12*^ 
grade level, r(25)= -.44, p<.01, suggesting that low and medium performing students decreased instances of 
unexcused tardiness from 10^^ to 12^^ grade while their counterparts in the control did not. 



Average Unexcused Absences (2002-2004 Academic Years) 




CEC 

-^Non CEC 



Figure 2 - Unexcused Absences (2002-2004) 
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Averaged Unexcused Instances of Tardiness (2002-2004 
Academic Years) 
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Figure 2 - Unexcused Instances of Tardiness (2002-2004) 



Statistically significant correlations of note for the CEC Intervention group included a moderately strong 
relationship between gender and 12*^ grade GPA with r(25) = .53, < .01 that indicated girls performed consistently 
higher in terms of GPA than boys from 9^^ to 12^^ grade. Also a strong relationship was found between pre-CEC and 
12^^ grade GPA with r(25)=.847, p<.001, which indicates that there was little improvement in GPA for students prior 
to and after attending CEC. Another pair of interesting relationships were, ironically, a moderately strong positive 
relationships between pre-CEC GPA and an increase in unexcused 12^^ grade absences, r(25)= .52, p<.01, and also 
unexcused instances of tardiness, r(25)=.41, p<.05), which suggest an increase in both for higher performing 
students. Another correlation which presents a different way of looking at this same relationship is that there was a 
moderately negative relationship between performance level and instances of unexcused tardiness at the 12*^ grade 
level, r(25)= -.44, p<.01 which suggests that low and medium performing students decreased instances of unexcused 
tardiness from 10^^ to 12^^ grade. A set of relationships where it is good to not have a strong relationship was 
between instances of unexcused absences and tardiness from the 10^^ to 12^^ grade level; in other words there was a 
general reduction of both across students with no significant relationship trends between those who had high 
instances of both at the 10*^ grade level as compared to the 12^^ grade. 

For the Non-CEC control group there were some similarities and also differences with the CEC 
intervention group in terms of statistically significant relationships amongst variables. Similarities included a 
moderately strong positive relationship between gender and 12^^ grade GPA, r(25)=.386, p<.05 where girls started 
with higher GPAs in 9^^ grade and ended with higher GPAs overall in 12*^ grade. Another similar relationship were 
the strong relationships between 9^^ grade GPA and 12^^ grade GPA, r(25)=.821, p<.01 again suggesting that student 
GPAs remained relatively static during this time. An interesting difference in the control group data, however, were 
that strong positive relationships were found between 10^^ grade and instances of unexcused absences and tardiness 
with 11*^ grade and 12^^ grade levels - for unexcused absences the correlation between 10^^ and 11^^ grade was 
r(25)=.741 and for 10^^ and 12^^ r(25)=.691 both at the p<.01. The pattern was similarly strong for unexcused 
instances of tardiness where the correlations between 10*^ and 1 1*^ was r(25)=.596, p<.01 and 10^^ and 12*^ grades 
was r(25)=.653, p<.01. 



41 



Discussion 



CEC represents an interesting opportunity to see what impact using system’s principles would have on a 
public school. In particular, this school represents a pilot test for the use of Harless’s ABCD system and the 
unveiling of his recent addition of a new “A”, assessment, to the traditional ADDIE model making it now A- 
ADDIE. Harless also wanted the CEC implementation to represent an example of needs-based as opposed to content 
based education. The results of this study suggests that CEC has used many aspects of system’s principles and had a 
positive impact on many fronts while at the same time being faced with a number of opportunities for improvement. 

Harless from a criterion referenced standpoint rated his overall satisfaction level with CEC a five out of 10 
or 50% level of satisfaction. He was also quick to point out that from a norm-referenced standpoint, in comparison 
to how other schools were doing, he would rate CEC much higher. Compared to his expectations, however, CEC is 
only half way there to adhering to the overall fidelity of his ABCD model and vision for what he wants it to become. 
His rationale included the fact that necessary comprises had to occur along the way that diluted the overall 
application of the ABCD system. During the Assessment phase (the first “A” in A- ADDIE) for example a 
misalignment occurred between the findings of the needs assessment, identification of specific skills associated with 
the jobs identified, and selection of the curriculum that was offered by CEC. Two primary reasons for this were that 

1) both high school and technical college courses already existed and existing curriculum is always easier to use and 

2) selection of curriculum occurred primarily using content and subject-based standards rather than behavior 
oriented, performance based skills. Three other major disappointments Harless had were the overall lack of training 
and preparation of the teachers with his instructional science principles (he received only 30 of the 80 hours of 
training needed), the lack of systematic follow up with graduates of CEC, and the minimal impact CEC has had on 
the local educational community in terms of educational reform. 

In many ways, however, it appears that system’s principles are working, especially in the front Assessment 
and Analysis phases and the back end Evaluation phase. By conducting a comprehensive needs assessment over a 
two year period not only were the specific job needs of employers identified, which surprisingly emphasized the 
need for soft skills (i.e. work ethic) over hard skills, most importantly a level of buy-in and cooperation amongst 
major stakeholders took place to collectively address the problem of education. As a result of this collaboration, the 
decision to use charter school status was made that not only gave CEC the freedom to innovate but also served to 
create the economic conditions that made the school a true joint-venture between major partners, all of whom share 
the financial benefit of a CEC student. This is a major difference between CEC and most other charter schools - it is 
a non-immersion school whose students belong to one of three base high schools and at times also are shared by a 
technical college as well. This appears to have created a positive environment of collaboration as opposed to the 
traditional competitive conditions focused on district wide accountability charter schools are typically associated 
with and general stigma going exclusively to a technical school traditionally has. Given the desired outcomes of 
employers and an infrastructure built around consensus and collaboration, CEC then garnered legislative and 
financial support from the highest echelons of Georgia state government - direct support from the Governor himself 
- and was provided over $14 million in initial funding and capital support (including the building and land CEC was 
built upon). Prior to opening its doors CEC appeared to be built on a solid foundation for success. 

The mission of CEC became “To create a viable 2V^ Century workforce” and it generated tremendous 
amount of local and statewide support and recognition anointed by the Governor as the model for technical 
educational reform for the state. Momentum was strong. Yamaha even decided to remain and expand its golf-cart 
manufacturing plan, at an estimated initial economic impact of $75 million to the local economy, because it felt that 
the presence of CEC would provide the kind of local skilled labor it needed to be successful. Six years later, what 
can be said about what we have learned from CEC about the use of system’s principles in public education and what 
impact has it had on students? 

The apparent success of CEC suggests that the use of system’s principles, especially given the rise of 
charter schools, provides an opportunity to take major tenets of performance technology - such as being customer 
and needs-based focused - and apply it successfully to public education. CEC represents the application of system’s 
thinking more from a macro rather than micro perspective with greater fidelity to the A- ADDIE process in the front 
and back end stages wrapped around the curriculum of mostly existing technical education and the traditional talent 
and experience of professional career educators, the majority of whom have little or no training with Harless’s 
ABCD system or formal instructional design methodology. With this reality, however, is the recognition that both 
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the technical curriculum and the teaching methodology centered on teaching mainly courses focused on technical 
skills, appear to inherently adhere to major system’s principle tenets including clearly identified and articulated 
ends-based performance outcomes, hands-on practice, and a broad range of mixed methods of instruction. 

On the evaluation side of the A-ADDIE process, CEC through Dr. Harless approached FSU’s Instructional 
System’s program to help with evaluation during their first year of operation. Evaluation has been consistent and 
focused on both formative and summative areas ever since. Ultimately, using a system’s framework to guide the 
evaluation process, the decision was made to focus resources primarily towards the inner working of CEC as a 
living, breathing system. In other words, attempting to understand what CEC was and to what degree stakeholders 
were satisfied became the primary focus emphasizing Ghiradeghi’s (1999) system’s definition of function, structure, 
and process. In the past two years, however, the evaluation efforts have shifted steadily towards summative 
evaluation of end-results leading up to the creation of a viable, sustainable process for following graduates into their 
post-secondary lives. The scarcity of consistent feedback from graduates and businesses represent two major 
limitations to the current study due to significant constraints in time and resources in which to do so. 

In terms of empirical results, the charter objectives of CEC, all of which initially were district level 
measures, have been favorable. Drop out rates were reduced by over 60% district-wide in the first year CEC was 
opened and have remained low. Ironically, there has been also a spike in district-wide SAT scores close to a 100 
point increase also since CEC has opened its doors. With on average only 25% of the county high school students 
attending CEC each year, however, it is not possible to link either district-wide measure to CEC outcomes. But, as 
charter objectives nonetheless, CEC has consistently met them (CEC has since changed its charter objectives at the 
end of its fifth year to more measurable objectives). Stakeholder satisfaction has been consistently high on a 
longitudinal basis across students, administrators, and parents. The common theme being the hands-on approach 
emphasizing preparation of students towards a life beyond high school, which appear to be at odds with “traditional” 
high school academics that appear to more focused on preparing for college. The one negative consistently 
mentioned is the strict tardiness and attendance policy administered by CEC. In addition, CEC faculty have reported 
their lowest satisfaction ratings in its six year history (80% satisfied) and appear to be struggling with the high 
turnover rate of general administration and the role they play in helping establish and implement CEC’s mission and 
vision. The two primary areas of dissatisfaction appear to be with the dissonance occurring between specific 
educational administrators and a subset of faculty and a changing curriculum driven by student enrollment that is 
redefining the course offerings and has led to reevaluation of positions and, in some cases, even an apparent loss of 
jobs. 



Comparing intervention and control groups across a three year period suggests several trends. First, both 
groups in terms of GPA, remained consistent over time and improved little from 9^^ to 12^^ grade. This suggests that 
CEC as well as the three base high schools do not appear to improve academic performance in terms of grades 
beginning at 9^^ grade - student performance remains relatively static throughout their high school careers. Another 
interesting finding is the differences in performance between boys and girls both which were similar across groups 
with girls having a higher mean GPA than boys at the 9^^ grade level and maintaining this difference throughout 
high school. This could be caused by the fact that there were more boys than girls in each group or suggest a 
meaningful difference that needs more in depth research. The major impact CEC appears to be having on students 
that differentiates itself from the control group is the impact on the soft skills as measured by unexcused absences 
and instances of tardiness - CEC students showed a statistically significant drop in unexcused instances of tardiness 
from 10^^ to 12^^ grade and, most importantly, showed no significant correlations between overall absences or 
tardiness from 10^^, 1 and 12^^ grade meaning the major offenders at the 10^^ grade level improved over time in 
both areas. For the control group, however, overall absences and tardiness increased over time and there were strong 
correlations between major offenders in the 10^^, 1 and 12^^ grades suggesting their relative behavior experienced 
little to no change over time. 

There are three major limitations to this study. First, GPA by itself is not a strong measure of academic 
performance and thus, any conclusions must be tempered by the lack of a robust indicator of academic performance. 
The author is working with the school district to collect additional measures of academic performance, including 
graduation rate and standardized test scores, for all students in both groups. Second, the lack of data from both 
graduates and business stakeholders - not having this data minimizes the overall impact of this study’s results as 
both represent major stakeholders for CEC. At this time, only anecdotal data has been collected that does suggest 
general satisfaction for both groups. And, Third, small sample sizes in terms of both qualitative and quantitative data 
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collection. At this juncture, as Dr. Harless has pointed out, there has not been a consistent, established system for 
data collection across time that would generate the necessary results needed to ensure proper input from major 
stakeholders. Although consistent evaluation has occurred the overall evaluation design has not been well integrated 
into the day-to-day operations of the school. 

CEC has begun its seventh year as a 2004 model high school award winner and recognized model for 
educational reform for the state of Georgia. Three separate dissemination efforts are underway currently in Georgia 
and the model is receiving national attention as well. The results of the study suggest that system’s principles have 
indeed been used effectively in the design, implementation, and evaluation of the school to some level of success - 
major stakeholders appear satisfied, what appears to be a significant impact on soft skills across time, and 
continuous emphasis on improvement and evaluation. This success suggests that indeed, given the right 
circumstances and support, that system’s principles can be used effectively, especially given the rapid growth of 
charter schools nationwide. Despite CEC’s current success, opportunities for improvement abound including the 
need to address gaps in its research and evaluation system that follow graduates into the post-secondary world, 
working with faculty to help reconnect them to the mission and goals of CEC given a changing curriculum defined 
by student enrollment and an administration characterized by high turn over, and the need to continuously improve 
its operations especially so that it continues to be deserving of its model school status and can support efforts by 
others to replicate this model. 



44 




References 



Berliner, D. & Biddle, B. (1996). Making Molehills Out of Molehills: Reply to Lawrence 
Stedman 's Review of The Manufactured Crisis. Education Policy Analysis 
Archives, Volume 4, Number 1, 1996 

Betts, F. (1992). How Systems Thinking Applies to Education. Educational Leadership, 

Nov. 1992,38-41. 

Gharajedaghi, J. (1999). Systems Thinking: Managing Chaos and Complexity: A 

Platform for Designing Business Architecture. Oxford: Butterworth-Heinemann 

Harless, J. (1997). The Eden Conspiracy: Educating for Accomplished Citizenship. 

Wheaton, IE: Guild V Publications. 

Hawley, W. (1988). Missing Pieces of the Educational Reform Agenda: Or, Why the 

First and Second Waves May Miss the Boat. Educational Administration Quarterly, 24 (4), 416-437. 

Husselkamp, R., Carson, C., and Woodall, T. (2000). Revisitng the Sandia Report. New Mexico Middle School 
Journal, Spring 2000, pg. 43-46 

Meier, D. (1998). Can the Odds Be Changed? Phi Delta Kappan, Jan. 1998, 358-362. 

Ravitch, D. (2000). Left Behind. New York, NY: Simon & Schuster 

Reigeluth, C. (1993). The Imperative for Systemic Change. In Systemic Change in 
Education (pp. 3-11). Englewood Cliffs, NJ: Educational Technology 
Publications 

Stedman, D. (1996). The Achievement Crisis is Real: A Review of The Manufactured 
Crisis. Education Policy Analysis Archives, Vol. 4, No. 1, 1996 

The National Commission on Excellence in Education (1983). A Nation at Risk: The 
Imperative for Educational Reform. Retrieved on January 12^^, 2005 from 
http://www.ed.gov/pubs/NatAtRisk/title.html . 

Thomas B. Fordham Foundation (1999). A Nation Still at Risk (Report No. EDO-TM-99- 
02). Washington, DC: ERIC Clearinghouse on Assessment and Evaluation. 

Waters, G. (1998). Critical Evaluation for Education Reform. Education Policy Analysis 
Archives, Vol. 6, No. 20, 1998 



45 



Tool Use in Open Learning Environments: 

In Search of Learner-related Determinants 

Geraldine Clarebout & Jan Elen 
Center for Instructional Psychology and Technology 
Katholieke Universiteit Leuven 

Introduction 

Various established researehers make the elaim that open learning environments (OLEs) may foster the 
aequisition of eomplex problem solving or higher order reasoning skills (Jaeobson & Spiro, 1995; Jonassen, 
1997). OLEs typieally eonfront learners with problems that they have to investigate from different perspeetives 
in order to propose an adequate solution. Considering multiple perspeetives is argued to stimulate students’ 
eognitive flexibility whieh in turn enables students to deal more easily with new eomplex problems (Krems, 
1995; Spiro, Feltovieh, Jaeobson, & Coulson, 1991; Jonassen, 1999). Jaeobson and Spiro (1995) report a study 
in whieh students eonfronted with a eomplex problem in an OLE (hypertext environment) perform better on a 
transfer task than students who were eonfronted with the same problem in a eomputer-based drill and praetiee 
format. These findings are explained by referring to inert knowledge aequired by learners in the ‘elosed’ 
learning environment eondition. To avoid sueh ‘funetional fixedness’, Anderson (1990) proposes to introduee in 
OLEs problems in all their eomplexity. Several other researehers (Barab & Duffy, 2000; Honebein, Duffy, & 
Fishman, 1993; Spiro et al., 1991) have also suggested that problems in OLEs should be as eognitively 
demanding as problems in the ‘real world’. 

Apart from a eomplex authentie problem, these OLEs are typieally eharaeterized by a large amount of 
learner eontrol. The environment eneourages learner-initiated enquiry and manipulation of the learning 
environment (Hannafin, Hall, Land, & Hill, 1994). The exploration and manipulation of the different 
eomponents of the learning environment also entails deeiding on the use of different support tools. More 
speeifieally, in order to reeeive support in OLEs students have to deeide themselves when the use of a speeifie 
tool eould be benefieial to their learning. In other words, the provision of OLEs rests on the assumption that 
learners are high self-regulators. While this might be the ease for speeifie groups of learners, researeh on learner 
eontrol (Friend & Cole, 1990; Goforth, 1994; Large, 1996; Williams, 1996) reveals no elear benefit of learner 
eontrol on learning. The main reason exaetly is that students are not always eapable of making adequate 
deeisions with respeet to their learning proeess. 

Researeh on tool use eorroborates this finding. Students indeed tend to disregard or inadequately use 
support deviees that are put to their disposal (Aleven, Stahl, Sehworm, Fiseher & Wallaee, 2003; Clarebout & 
Elen; in press). If students do not or inadequately use tools and are not eapable of making adequate deeisions, 
doubts raise about the effeetiveness of OLEs. 

In view of the design of effeetive OLEs it is important to gain more insight in the variables that 
influenee tool use, more preeisely learner-related and environmental faetors. With respeet to the environmental 
faetors some studies have already been done. For instanee, the influenee of adviee on tool use has been 
addressed by Lee & Lehman (1993) and by Grasel, Fiseher, and Mandl (2001). Both studies reveal a benefieial 
effeet of adviee on tool use, i.e. the frequeney of tool use inereased for those students reeeiving adviee versus 
those students who did not reeeive adviee. Another environmental faetor studied was instruetional method. 
Crooks, Klein, Jones, and Dwyer (1996) eompared a group of students working individually on a task and a 
group of students working eo-operatively. They found that students working individually more frequently used 
tools than the eo-operative groups. Unfortunately, Crooks, Klein, and Savenye (1998), eould not replieate their 
findings. 

This eontribution foeuses on learner-related variables, more speeifieally metaeognitive variables. The 
impaet of metaeognitive skills, sueh as monitoring and regulation, on tool use has not yet been explored, despite 
the theoretieal relevanee of these variables (Clarebout & Elen, in press). Other student-related variables were 
already studied, but overall no elear effeets were found (for reviews see: Aleven et al., 2003; Clarebout & Elen, 
in press). Some studies for instanee, report an effeet of prior knowledge and ability (e.g., Viau & Larivee, 1993), 
while others do not find sueh effeet (e.g., Sehnaekenberg & Sullivan, 2000). The same is true for learning style, 
where some do (e.g.. Carrier, Davidson, Higson, & Williams, 1984) others (e.g., Lee & Lehman, 1993) do not 
find an effeet. 

This eontribution investigates the impaet on tool use of three metaeognitive variables: students’ 
regulation aetivities, help-seeking behavior, and instruetional eoneeptions. In line with Flavell (1979) and 
Vermunt (1992), regulation aetivities are aetivities students undertake to regulate, monitor and eontrol their own 
eognitive proeesses. Regulation aetivities eomprise the applieation of eognitive and environmental resourees as 
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demanded by the task (Newman, 1998). It is hypothesized that students’ regulation skills will have a positive 
influenee, resulting in inereased tool use in OLEs. 

Newman (1991, 1994) defines help seeking behavior as effieiently seeking for assistanee when a laek 
of eomprehension is pereeived. Help seeking also ineludes the ability to identify one’s own problem, i.e. of 
monitoring one’s own knowledge (Sehunk & Zimmerman, 1994). Aleven et al. (2003) suggest that help seeking 
behavior is related to tool use. The more students show help seeking behavior, the more they will be inelined to 
use tools. Or, the more students tend to avoid help, the less inelined they will be to use tools. The literature on 
help seeking (Newman, 1991; Pajares, Cheong & Oberman, 2004) also eonsidered the pereeived benefits of help 
seeking as an aspeet of, or as being related to, help seeking behavior. In his study, both help avoidanee behavior 
and pereeived benefits of help seeking are studied in relation to tool use. 

Students’ instruetional eoneeptions have been deseribed as a partieular kind of metaeogntive 
knowledge (Elen & Lowyek, 1999). They are defined as ‘all ideas, eoneepts, and theories that an individual 
learner holds about (eomponents of) a learning environment.’ (Lowyek, Elen & Clarebout, in press). Winne 
(1985, in press) states that the funetionality students aseribe to a tool, will determine whether, and how they use 
this tools. Henee, it ean be assumed that instruetional eoneeptions influenee tool use. A typieal example of the 
impaet of instruetional eoneeptions is provided by Andre (1979). In this study, pre-questions were provided to 
students to give examples of test items. But rather than using these items as examples of test, students used them 
to determine what was important in the eourse book. As sueh, it is hypothesized that the more students eoneeive 
the tools as funetional to problem solving, the more they will be inelined to use them. 

Figure 1 provides the model that will be tested in this eontribution. Sinee all independent variables 
relate to a speeifie part of metaeognition, they might be eorrelated. The lines between the independent variables 
illustrate these possible eorrelations. 



Regulation 



+ 



Avoidance of 
help seeking 



Perceived 
benefits of 
help seeking 



Conceived 

functionality 



Figure 1 : Sehematieal model 

Methodology 

This study was part of a broader researeh projeet in whieh the effeet of adviee on tool use was studied. Students 
in the study were randomly assigned to one of three groups: a eontrol group without adviee (C-group), an 
experimental group with adapted adviee (AA-group) and an experimental group with fixed adviee (FA- group) 
(Clarebout & Elen, 2005). Sinee these groups differ with respeet to this instruetional intervention (kind of 
adviee), the influenee of learner eharaeteristies will be studied in eaeh of these groups separately and eompared 

to one another. 

Participants 

Partieipants were 185 first year edueational seienees students. Partieipation was voluntary, but 
partieipants eould earn 10% of their grades for an introduetory eourse on ‘Learning and Instruetion’ when they 
partieipated. 

Materials 



Tool use: 
^ Frequency/ 
Time 
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Study materials. A computer-based learning program, called STUWAWA, was developed using Maeromedia 
Direetor. STUWAWA eonfronts student with an ill-struetured problem: the seleetion of the most environmental- 
friendly drinking eup for a musie festival, by eonsidering eeologieal, finaneial and safety aspeets. In view of 
inereasing authentieity, this eomplex problem was introdueed on video by a member of a neighborhood 
eommittee requesting the students for help to deal with the problem. STUWAWA eontains different types of 
tools, namely information resourees (e.g., information list; videos), performanee support tools (e.g., ealeulator), 
and knowledge monitoring tools (e.g., problem solving seript). The different tools provide aeeess to information 
and support students’ eognitive proeesses. Students deeided themselves whether or not to use the tools. Figure 2 
provides a sereenshot of the learning environment (see Clarebout & Elen, 2004 for a detailed deseription). 




Figure 2: Sereen dump of STUWAWA 



Regulation activities. Students’ regulation aetivities were assessed with the metaeognition items of the Learning 
Style Inventory (LSI) of Vermunt (1992). The LSI is a widely used instrument to measure students learning 
styles (e.g., Boyle, Duffy, & Duleavy, 2003; Sehouwenburg, 1996; Veenman, Prins, & Verheij, 2003). Students 
were asked to indieate on a six-point Likert type seale their level of agreement (1 = totally disagree; 6 = totally 
agree) with 28 statements of the LSI relating, to students’ regulation aetivities. These 28 items measure three 
types of regulation (Vermunt, 1992): self-regulation (e.g.. After eaeh paragraph I try to formulate the learning 
eontent in my own words to test my learning progress.), external regulation (e.g., I study aeeording to the 
instruetions given in the study materials or provided by the teaeher.), and laek of regulation (e.g., I notiee that it 
is diffieult for me to determine whether I master the subjeet matter suffieiently.). 

Help seeking behavior. For help seeking behavior the instrument of Pajares et al. (2004) was translated in Duteh 
and adapted to the eontext. This adaptation involved replaeing ‘eomputer seienee problems’ by ‘solving 
problems’. Students reported the extent they agreed with the different statements on a six-point Likert type 
seale. The instrument eonsists of 9 items measuring students’ help avoidanee behavior (e.g., I would write down 
any answer rather than ask for help in elass.) and 6 items measuring students pereeived benefits of help seeking 
(e.g., I think asking questions in this elass helps me learn.). 

Instructional conceptions. The ICON-questionnaire was used to determine students’ instruetional eoneeptions 
(Clarebout, Sarfo, & Elen, 2004). Students were asked to indieate on a Likert- type seale to what extent they 
agreed that eertain tools were funetional to solving eomplex problems (e.g., Aeeording to me, a problem solving 
seript helps students to better understand the eontent.). For eaeh tool, a short explanation was given of the tool’s 
funetionalities. 

Log files. To register students’ tool use log files were kept in an Aeeess database. This allowed to register 
students’ aetions in an unobtrusive way (Jaekson, 1990; Young & MeNeese, 1995). The database eontains 
students’ identity, all elieks they made in the program, the time when they elieked on a tool, the starting time 
and the ending time. 

Procedure and Design 

During the first elass of the eourse on Learning and Instruetion, the different instruments measuring 
learner eharaeteristies were administered. Students were asked to put their name on the questionnaire in view of 
linking the questionnaire to the log file data. In a seeond session, students worked individually with 
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STUWAWA. The introduction for this second session was identical for all participants. The researcher 
explained that participants would have to solve a problem on the computer, that they had maximum one hour to 
solve the problem and that after that hour they had to deliver their solution. Students could work at their own 
pace for a maximum of 1 hour. When students finished they were kindly thanked for their participation and 
asked to not inform the others about the content of the problem. 

Data analyses 

To ascertain the influence on tool use, two dependent variables were considered, namely frequency of 
tool use, and the proportion of the total time spent on tools. The frequency of tool use is the sum of all clicks 
students make to access a specific tool. The total time proportionally spent on tools is the sum of the time spent 
on all tools divided by the total time spent on the problem. 

For the different instruments, first reliability analyses were performed using the internal consistency 
measure Cronbach Alpha. In line with Fraenkel and Wallen (2003), an alpha- value of .70 will be considered as 
representing a good internal consistency. Scales were constructed by summing up the scores on the different 
items and dividing the sum by the number of items. These scales were used in further analyses. First, 
correlations between the independent variables were calculated to check whether there was a problem of 
multicollinearity between the independent variables (Tacq, 1997). Since the four independent variables are all 
assumed to measure an aspect of self-regulation, multicollinearity might indeed constitute a problem, and should 
be controlled for (Cohen, Cohen, West & Aiken, 2003; Tacq, 1997). Second, it was checked through a 
MANOVA whether the three groups, the AA-, FA- and control-group, differed with respect to the independent 
variables. Next, regression analyses were done separately for each group and for each dependent variable. For 
these regression analyses, all different models were explored for all three groups: first models with the four 
independent variables were entered, next models with combinations of three independent variables, then models 
with combination of two independent variables and finally for each independent variable separately. This 
allowed to identify for each group the best model, looking at the R^, and to gain more insight in the role these 
independent variables play in the model. The t- values for the different variables provide information on the 
contribution of each independent variable to the model. This method was opted for rather than a ‘stepwise 
regression analysis’, since problems have been reported (such as increased Type-I errors) when using stepwise 
regression for theory testing (Menard, 1995; Cohen et al. 2003). Additionally, the VIF-index was looked at as a 
double check for multicollinearity (Cohen et al., 2003). 

Results 

Overall, the internal consistency analyses (see Table 1) show good reliabilities for the self-regulation-, 
the help avoidance-, the perceived benefits of help seeking-, and the instructional conceptions-scale. Hence, 
these scales will be used in further analyses. Two LSI-scales, namely the external regulation and the lack of 
regulation scale, show weaker reliabilities, and are not used in further analyses. 



Table 1: Cronbach Alpha for LSI-scales, Help Seeking 
Behavior Scales, and the ICON-scale 



Scale 


Cronbach 

Alpha 


Learning Style Inventory 




Self-regulation 


.80 


External regulation 


.62 


Lack of regulation 


.68 


Help seeking behavior 




Help avoidance behavior 


.90 


Perceived benefits of help seeking 


.86 


ICON-questionnaire 




Conceived functionality 


.91 



The correlation matrix for the independent variables is presented in Table 2. Some correlations are 
significant. A negative correlation is found between avoidance of help seeking behavior and the perceived 
benefits of help seeking behavior in the AA and FA-group. In the FA-group, an additional positive correlation is 
found between the conceived functionality of tools and self-regulation. In the control-group no significant 
correlations were found. 
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Since, these eorrelations are rather low, no problem with multieollinearity of the independent variables 
is expeeted, unless the toleranee values or the VIF index indieated otherwise. 



Table 2: Correlation Matrix for the Independent Variables for the Three Groups 





HA 


BH 


CF 


SR 


AA-group 


HA 


1.00 








BH 


-.329* 


1.00 






CF 


-.024 


-.002 


1.00 




SR 


-.175 


.101 


.162 


1.00 


FA-group 


HA 


1.00 








BH 


-.285* 


1.00 






CF 


-.196 


.015 


1.00 




SR 


-.238 


.158 


.428** 


1.00 


C -group 


HA 


1.00 








BH 


-.027 


1.00 






CF 


-.203 


-.141 


1.00 




SR 


-.261 


.121 


.214 


1.00 



HA = help avoidanee; BH = pereeived benefit of help seeking behavior, 
CF = eoneeived funetionality; SR = self-regulation 



The MANOVA with the different groups (AA, FA, and eontrol) as independent variables and the four 
learner eharaeteristies as dependent variables did not show any signifieant effeet = .957; F(8,332)=.923; p 
>.05; eta^ = .02). This means that on average the groups do not differ with respeet to these learner 
eharaeteristies. While based on the eorrelations between the variables one eould argue that the groups do differ, 
none of these eorrelations differ signifieantly between the groups. 

In a first step of the regression analyses, for all three groups a full model was tested, i.e. it was studied 
how mueh of the varianee in the dependent variable eould be explained by the four independent variables in 
eaeh group. The results of the regression analyses for the three groups are presented in Table 3. 

In a first seetion, the results are diseussed with respeet to frequeney of tool use, in the seeond seetion, 
those with respeet to the proportion of time spent on tool use. 
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T able 3: Regression Outcomes for the Three Groups with Four Independent Variables (IV) 



Group 


R" 


IV 


Beta 


t-values 


DV= frequency of tool use 


AA-group 


.019 


SR 


.023 


.164 






HA 


-.009 


-.060 






BH 


.074 


.521 






CF 


.106 


.776 


FA-group 


.14* 


SR 


.170 


1.232 






HA 


.058 


.436 






BH 


-.113 


-.869 






CF 


-.384 


-2.773* 


C -group 


.045 


SR 


-.007 


-.049 






HA 


.147 


.957 






BH 


.115 


.792 






CF 


-.075 


-.504 




DV= 


proportion of time spent on tool use 




AA-group 


.10 


SR 


-.016 


-.117 






HA 


-.080 


-.576 






BH 


-.258 


-1.886 






CF 


-.204 


-1.560 


FA-group 


.12 


SR 


-.203 


-1.448 






HA 


.275 


2.042* 






BH 


.050 


.375 






CF 


.029 


.204 


C -group 


.053 


SR 


-.168 


-1.100 






HA 


-.090 


-.591 






BH 


.164 


1.131 






CF 


.087 


.585 



* significant on .05-level 



Frequency of Tool Use 

The analysis reveals a significant model for frequency of tool use for the FA-group. This model 
explains 14% of the variance in frequency of tool use. The model shows that in the case of fixed advice self- 
regulation and help avoidance contribute positively to frequency of tool use, while the perceived benefits of help 
seeking and the conceived functionality contribute negatively. The t- values of the independent variables reveal 
that mainly the conceived functionality predicts frequency of tool use. The more students conceive tools as 
being functional, the less frequent they use them in the group with fixed advice (FA-group). For the other two 
groups no significant model, nor a significant predictor for frequency of tool use was retrieved. 

In the consequent regression analyses, models with three, two and one independent variable were tested 
to gain more insight in the effects of the different independent variables. For the control group and the adapted 
advice group no better models could be found for frequency of tool use. 

For the fixed advice group, it was found that when leaving out the help avoidance variable, the model 
still explains 14% of the variance in the dependent variable. Self-regulation activities (beta = .166, t = 1.209) 
and the perceived benefit of help seeking behavior (beta =-.130; t = -1.053) are not significant, but removing 
them from the model, reduced the to .09. The conceived functionality variable in the SR-BH-IC model had a 
beta- value of -.394 and a t- value of -2.898. The F-test to check for significant differences between the R^ values 
of the different models (Cohen et al., 2003), did not receive any significant difference between the 4, 3 or 1- 
variable model (F = 1.071; p >.05). From these results it can be concluded that especially students’ conceived 
functionality predicts the frequency of tool use. However, removing self-regulation and perceived benefits of 
help seeking behavior reduced significantly the predictive value for the conceived functionality. As such, a 
model for the fixed advice group, including these variables leads to a better prediction of frequency of tool use 
than one without these ‘non-significant’ variables. 

Proportional time spent on tools 

For the proportional time spent on tools, no model reached significance, but for the fixed advice-group 
a significant effect was found of help avoidance behavior. This variable contributed positively to the 
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proportional time spent on tools. Meaning that the more students indieate to avoid help seeking behavior, the 
more time they spent on tools. 

With respeet to the time spent on tools, no signifieant model eould be found for the AA and C-group by 
leaving out eertain variables. However, an almost signifieant (p = .06) effeet is found of pereeived benefits of 
help seeking when leaving out the self-regulation variable for the adaptive adviee-group. This result suggests 
that the more students pereeive help seeking as benefieial the less time they spent on tools. For the fixed adviee- 
group a signifieant model was found with three independent variables (R^ = .13), when removing the eoneeived 
funetionality variable (see Table 4). Deleting the pereeived benefits of help seeking variable in this model gives 
the same R^- value (Table 4). The regression eoeffieient for self-regulation aetivities and help avoidanee did not 
ehange signifieant when removing the other variables from these models. The model indieates that the less 
students are inelined to engage in self-regulation aetivities and the more they indieate to avoid help seeking 
behavior, the more time they spent on tools. 

For none of the models, the toleranee value and the VIF-index revealed problems with eollinearity; and no 
problems eould be found with nonlinearity of the data. 



Table 4: Two Models of the FA-group for Time Spent on Tools 



Group 


R" 


IV 


Beta 


t-values 


FA-group 


.13 


SR 


-.201 


-1.606 






HA 


.269 


2.061 






BH 


.040 


.308 




.13 


SR 


-.198 


-1.596 






HA 


.258 


2.077 



Discussion 



This eontribution investigated the influenee of different metaeognitive learner eharaeteristies 
on tool use. Inereased tool use was hypothesized to oeeur in ease of higher engagement in self-regulation 
aetivities, lower help seeking avoidanee, higher beliefs in benefieial effeets of help seeking behavior, and higher 
eoneeived funetionalities of tools. Surprisingly, none of these hypotheses eould be eonfirmed, and the 
theoretieal model was totally falsified. For the adaptive adviee-group and the eontrol group a good fitting model 
for frequeney of tool use eould not be found. Even worse, none of the theoretieally relevant independent 
variables signifieantly eontributed to the varianee in the frequeney of tool use. For the proportional time spent 
on tools, an almost signifieant effeet was found of pereeived benefits of help seeking in the adaptive adviee- 
group. Interesting in this ease is the sign of the beta-value. Apparently, the more students pereeive help seeking 
as benefieial the less time they spent on the tools. The items in the seale may provide an explanation. All items 
in the seale on pereeived benefits of help seeking relate to asking questions to the teaeher or to peers. It looks 
that the more students are inelined to ask for help from humans, the less they are inelined to use support tools. 
Therefore, this result ealls for a study in whieh this human-effeet would be direetly tested. 

While for the adapted adviee- and the eontrol-group a signifieant model eould not be retrieved, a 
signifieant model was found for the fixed adviee-group. For frequeney of tool use, the best model ineludes 
pereeived benefits of help seeking, self-regulation aetivities and eoneeived funetionality as independent 
variables. Interestingly, the signs of the independent variables’ eontributions are onee more opposite to what 
was expeeted for the pereeived benefits of help seeking and eoneeived funetionality. The more students posses 
self-regulation skills and the less they pereeive help seeking behavior and the tools as funetional the more 
frequent they use these tools. Espeeially eoneeived funetionality plays a signifieant role. A possible explanation 
eould be that the adviee given on the speeifie tools eauses a eonfliet with students' own eoneeptions about why 
they would think these tools are funetional to their own problem solving proeess. Peeuliar is the signifieant 
positive eorrelation found in this group between students’ self-regulation skills and their eoneeived funetionality 
of tools. This positive eorrelation would suggest that the more students possess monitoring skills, the more they 
ean see the funetionalities of tools. However, in the retrieved model, self-regulation eontributes positively to 
frequeney of tool use, while eoneeived funetionality eontributes negatively. This result ealls for more in-depth 
theoretieal work on the relationship between self-regulation and tool use. 

In any ease, results for the eontrol, adapted and fixed adviee group reveal problems with respeet to 
predieting aetual study behavior based on self-reporting instruments only. As suggested by Winne (in press) 
detailed observations of aetual student behavior through log-files may help to resolve a number of the issues 
involved. 

In the fixed adviee-group a fitting model was found for time spent on tools, ineluding two of the four 
independent variables, namely self-regulation skills and help avoidanee behavior. Onee more, some unexpeeted 
results were found. The more students indieate to avoid help, the more time they spent on tools. The more 
students possess self-regulation skills, the less time they spent on tools. With respeet to the help avoidanee 
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behavior, a similar remark ean be made as above with respeet to the human-effeet. In the questionnaire the items 
measuring help avoidanee behavior were related to students’ behavior in elass. Students’ who avoid asking for 
help in their elass, may be more inelined to first look for help in the material at hand, in this ease, the eomputer 
program and explore all possible information that is available. For that reason, they might have weleomed the 
adviee that was systematieally given every seven minutes. Looking at the literature on help-seeking (e.g., 
Newman, 1998), this indeed deals mainly with asking other humans for help. This form of help-seeking is 
probably a more eomplex behavior than merely eonsulting tools, involving not only eognitive aspeets, but also 
motivational aspeets (Newman, 1998). It has been found that asking for help is often seen as admitting ones 
failures (Valentine, 1993). Students’ who hold this pereeption tend not to ask help from teaehers, but more 
frequently eonsult informal resourees (Keefer & Karabeniek, 1998). Looking at tools as informal resouree of 
help eould explain the retrieved influenee of avoidanee of help seeking on tool use. Students who avoid asking 
help from teaehers, try to find a solution in the tools, the informal resourees. 

Overall, the results reveal mixed results with respeet to the influenee of metaeognitive variables on tool 
use. It remains however eurious how the groups differ with respeet to the models retrieved and the eontribution 
of the different independent variables. It seems that depending on the presenee and kind of adviee, different 
learner eharaeteristies play a role in the frequeney of, and proportional time spent on tool use. Moreover, these 
student eharaeteristies often eontribute to tool use in an opposite way then was expeeted. 

Conclusion 

This study revealed that, even though for one group a signifieant model was found, a large pereentage 
of varianee in the dependent variables was not explained, indieating that there must be a number of additional 
variables that should be eonsidered in order to explain frequeney of and proportional time spent on tool use. 
While from a theoretieal point of view metaeognitive variables seem important to eonsider, others seems even 
more important. Clark, Howard and Early (in press) for instanee, point to motivational aspeets involved in 
solving eomplex problem solving. They state that motivation aeeounts for almost as mueh learning varianee as 
eognitive aptitude. Motivation is defined as self-effieaey, emotional regulation and attribution of mistakes. 

Additionally, rather than students’ instruetional eoneeptions, Aleven et al. (2003) suggest that students’ 
epistemologieal beliefs may influenee students’ tool use. A preliminary indieation that there may indeed be a 
relation between epistemologieal beliefs and tool use is provided by Hartley and Bendixen (2001). They indieate 
that, eontrary to what they expeeted, signifieant positive eorrelations were found between naive epistemologieal 
beliefs and tool use. However, although signifieant, the eorrelations were rather low ( <.3 00). 

Furthermore, it may be wondered what the role is of prior knowledge. Prior knowledge has been shown 
to be an important predietor of learning results (Doehy & Alexander, 1995). Of eourse, in eomplex problem 
solving, the question beeomes prior knowledge of what. Is it prior knowledge about the eontent of the problem, 
about problem solving, about the learning environment? The study ealls therefore for a more systematie analysis 
of the different types of prior knowledge that may affeet aetual student behavior in OLEs. 

The results of this study reveal that not only student eharaeteristies are important variables to eonsider, 
but also instruetional interventions. Apparently the instruetional intervention, e.q., adviee, ean moderate the 
effeet of student eharaeteristies. This implies that the effeet of an instruetional intervention is not only 
moderated by student eharaeteristies, but that the influenee of speeifie student eharaeteristies ean also be 
moderated by a speeifie instruetional intervention. 
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Abstract 

The purpose of this study was to describe preservice teachers’ experiences as they attempted to integrate technology 
into their student teaching experience. Furthermore, we explored the impact cooperating teachers have on 
preservice teachers during these field experiences. This study followed the experiences of eight preservice teachers 
from a liberal arts Christian university in the mid-south and their cooperating teachers during the fall of 2005. Data 
were gathered through self-report reflections, interviews and surveys. Results indicated preservice teachers were 
using technology to enhance lessons during their student teaching experience, their cooperating teachers’ support did 
affect their use of technology and preservice teachers were able to identify barriers and enablers which influenced 
their use of technology for teaching and learning. 

The purpose of this study was to examine experiences preservice teachers, at a private, liberal arts Christian 
university in the mid-south, encountered as they attempted to integrate technology in their student teaching. This 
study attempts to fill a gap in the literature by specifically exploring the support given by cooperating teachers as 
preservice teacher attempt to use technology in the classroom. This study will allow teacher educators to gain a 
better understanding of the range of challenges faced by preservice teachers, explore what effects the support of the 
cooperating teacher’s has on the preservice teachers’ training infrastructure, and empower preservice teachers to 
identify and deal with some of the barriers they may face, as they attempt to teach with technology. 

Research Questions 

This study was conducted to answer the following research questions: 

RQl: In what ways do preservice teachers use technology in teaching and learning, during the student teaching 
experience? 

RQ2: In what ways do cooperating teachers support preservice teachers’ uses of technology in the classroom? 

RQ3: What are the enablers and barriers for preservice teachers to integrate technology? 

Method 

Design 

With these questions, a qualitative approach to identify emerging themes or patterns across participants and 
overtime was needed. Qualitative data were collected to understand the preservice teachers’ use of technology, to 
identify the cooperating teacher’ s means of support, and to recognize barriers the preservice teachers experienced, 
using a case study approach. Pseudonyms have been used throughout the manuscript. A case study approach 
provides an opportunity to examine the uniqueness and commonality of the persons being studied and provides a 
chance to look at the interactions within its context in detail (Patton, 2002). 

Setting and Participants 

The study began with nineteen preservice teachers and their cooperating teachers from a private, liberal arts 
Christian university in the mid-south enrolled in student teaching in the fall of 2005. The preservice teachers had 
similar technology training and experiences, but their school placements and their cooperating teachers’ 
backgrounds and experience were very diverse. Therefore, purposeful, heterogeneous sampling was used (Patton, 
2002). The preservice teachers were seeking state licensure in early childhood, elementary, middle school, high 
school. Teaching English as the Second Language (TESL) and special education. Eight of these nineteen 
volunteered to participate in the study. Their respective cooperating teachers agreed to participate, as well. The 
licensure areas represented were elementary, middle school, TESL and special education. The preservice teachers 
were all white, Caucasian females whose ages ranged from 21 years to 43 years with a average age of 25 years. The 
cooperating teachers were employees of the local or nearby school districts. The cooperating teachers were all 
female Caucasians whose ages varied from 31 to 57 years with a mean of 43 years. Many of the cooperating 
teachers were considered veteran teachers. Their years of teaching ranged from five years to 27 years with a mean 
of 17 years. 
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Instrumentation 

Instruments used to collect data for the study included a weekly reflection, interviews and two surveys. 
Each of these is described below. 

Reflections. The reflection guide highlighted the specific technologies the preservice teachers used each 
week. Specifically, the preservice teacher were asked to identify whether they had used production tools, authoring 
tools, concept mapping applications, individualized tracking, and additional technologies (e.g., overhead projector, 
TV- VCR or DVD, scanner, SmartBoard, digital camera, other). For each technology used, the preservice teacher 
was asked to identify (a) where the technology-integrated lesson occurred (computer lab or with classroom 
computers) and (b) the pedagogical style of the lesson (teacher-directed or student-centered). In addition for each 
technology used, the preservice teacher was asked for a brief overview and any comments. After the first week, the 
reflection guide was modified, to distinguish between the teacher directed use of technology by the preservice 
teacher and the cooperating teacher. 

Interviews. Focus group interviews were used so that the participants could hear each other’ s responses and 
make additional comments beyond their original response based upon what others had to say. The interviews were 
scheduled for approximately an hour. The semi- structured interviews focused on the guiding interview questions 
that were aligned with the three research questions. The focus group interviews were audio/video taped and 
transcribed verbatim. The researcher made reflection notes as a part of the process. Five preservice teachers 
participated in the focus group interview. Because of their schedules and travel distances, one preservice teacher 
participated in an individual interview and two preservice teachers chose not to participate. Another focus group 
interview was scheduled for the cooperating teachers. Three cooperating teachers participated in the focus group 
interview. Two cooperating teachers participated in individual phone interviews, and a third participated in a face to 
face interview in her classroom, during the same week. 

Surveys. Two surveys were used with both the preservice teachers and the cooperating teachers. The 
Teacher Technology Questionnaire (Fowther & Ross, 2000) is a three-part instrument used to collect teacher 
perceptions of computers and technology. Only the first section was used with this research. In the first section, 
teachers rate their level of agreement with 20 statements regarding five technology-related areas: Impact on 
Classroom Instruction, Impact on Students, Teacher Readiness to Integrate Technology, Overall Support for 
Technology in the School, and Technical Support. Items are rated with a five-point Fikert-type scale that ranges 
from (1) Strongly Disagree to (5) Strongly Agree. Two primary questions are asked in the second section. The first 
asks teachers to rate their level of computer ability as very good, good, moderate, poor, or no ability. Next, teachers 
indicate if they have a home computer, and if they do, if they use the home computer to access instructional 
materials on the Internet and/or to prepare classroom materials. The Technology Skills Assessment (TSA) (Marvin 
et al, 2002) includes 50 three-point Fikert-type questions. The three-point questions are designed to assess the 
perceived technological abilities of the participants. All of the questions are arranged into seven categories, which 
are aligned to the International Society for Technology in Education’s (ISTE’s) National Educational Technology 
Standards (NETS) (Marvin, Fowther & Ross, 2002b). The categories of the survey are as follows.* Computer 
Basics, Software Basics, Multimedia Basics, Internet Basics, Advanced Skills, Using Technology for Learning, and 
Policy and Ethics. An additional section of the TSA that was not used in this research uses 6 multiple-choice 
questions designed to determine the actual knowledge of the participants on several areas of technological literacy 
also aligned with NETS. 

Data Anaylsis 

Qualitative data analysis. The qualitative data (i.e.., in-depth notes, reflections, interview transcripts) were 
analyzed using constant comparison and inductive analysis (EeCompte & Preissle, 1993) to guide the development 
of patterns and themes in the data. The first round in abstraction and data reduction began with manually sorting 
according to themes and open coding (Patton, 2002), based on commonly occurring words and phrases. Next, since 
the guided interview questions were aligned with the three research questions, the underlying themes (e.g., the 
preservice teacher use of technology in the student teaching experience) were exposed, thus a priori codes were 
applied to the data. Member checking, audit trail, peer debriefing and triangulation (Cresswell, 2003) were used to 
ensure trustworthiness. 

Quantitative data analysis. The quantitative data were gathered from the Technology Teacher 
Questionnaire and Technology Skills Assessment. An error was found in the TTQ, a question was repeated in two 
constructs. The question was removed from the constructs. An independent t-test was conducted to test the 
significance of the means between the preservice teachers and cooperating teachers for each item and then for each 
construct. With the large number of t-tests conducted, a Bonferroni adjustment was added. For the TTQ, the 
significance level was set to p<.002 (a=. 05/20), and for the TSA the significance level was set to /?<.001 (a=. 05/47). 
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Results 

To best represent these mixed method findings, the qualitative themes are only listed below without the 
thick descriptions. Next, the quantitative results from the two surveys are reported with their inferential and 
descriptive statistics. Then to situate these findings, the results as a whole are interpreted together in the following 
section with support from the qualitative and quantitative data, where the voices of the participants are presented. 

The underlying qualitative themes included: (1) use of technology, (2) cooperating teachers’ actual support, 
and (3) barriers and enablers experienced. Again, these three themes were based upon the research questions. The 
additional themes that emerged from the data included: (4) expectations for the use of technology and (5) technology 
guidelines. 

Surveys 

TTQ. The TTQ acknowledges the participants’ perception of technology. An independent- samples t-test 
compared the mean scores of the preservice teachers and the cooperating teachers using the TTQ. The mean and 
standard deviation were derived for each question and for each of the five constructs (i.e.. Impact on Classroom 
Instruction, Impact on Students, Teacher Readiness to Integrate Technology, Overall Support for Technology in the 
School, and Technical Support). Both the means and standard deviations for the preservice teachers and the 
cooperating teachers were calculated for comparison. Results indicated there was no significant statistical difference 
in the responses of the two groups on the individual questions. 

Interestingly, the mean (M=4.13) from both groups was the same on Question 12: My computer skills are 
adequate to conduct classes that have students using technology, thus the participants felt their computer skills were 
adequate to conduct classes that have students using technology. The greatest difference in the means was on 
Question 2: 1 can readily obtain answers to technology-related questions. The preservice teachers mean (M=3.38, 
SD=.744) much higher than the cooperating teachers mean (M=2.38, SD=1.188), indicated that the preservice 
teachers felt confident they could readily obtain answers to technology related questions. No statistical significance, 
however, was found. 

When the mean scores and standard deviations were derived for the five constructs, there was no statistical 
significance in the means of the preservice teachers and the cooperating teachers reflected on the TTQ in any of the 
constructs. 

TSA. The TSA revealed levels of confidence in the preservice teachers and the cooperating teachers’ use of 
technology. An independent t-test compared the mean scores of the preservice teachers and the cooperating teachers 
on each question, as well as for on each of the six constructs (i.e.. Computer Basics, Software Basics, Multimedia 
Basics, Internet Basics, Advanced Skills, and Using Technology for Learning). When the mean scores of the 
participants were compared on each of the 46 questions, there was a significant statistical difference on 1 of the 47 
questions. The mean and the standard deviation for Question 2: “Create basic computer documents (word processed) 
in a timely manner?” and Question 1 1 : “Print a document using “Print” form the File menu and/or the toolbar icon?” 
were the same for both groups. 

With each of the grouped constructs, the preservice teachers’ means were higher than the cooperating 
teachers’ means in every section. No significant differences were found with the constructs, however. 

Findings and Interpretations 

Three underlying themes based upon the research questions and guided by the interview questions and two 
additional themes emerged from the focus group and individual interviews with the preservice teachers and the 
cooperating teachers signaling issues that seem to affect preservice teachers’ integration of technology in the 
classroom during their student teaching experience. 

The underlying themes included: (1) use of technology, (2) cooperating teachers’ actual support, and (3) 
barriers and enablers experienced. Again, these three themes were based upon the research questions. The additional 
themes that emerged from the data, included: (4) expectations for the use of technology and (5) technology 
guidelines. Each of the five themes will be discussed in the section below and supported direct quotes from 
participants. 

Preservice Teachers Use of Technology in the Student Teaching Experience 

The first theme examined the preservice teachers’ actual use of technology in teaching and learning during 
the ten- week period. As the preservice teachers completed their reflections and their interview, the findings 
indicated all eight of the preservice teachers had access to technology. They used technology on a personal basis 
(e.g., e-mail, writing lesson plans) and all of the preservice teachers were able to use technology in the classroom in 
one form or another. 
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The findings of the present study were consistent with Grove, Strudler, and Odell’s (2004) report that 
classrooms teachers had access to technology. However, all of the current preservice teachers who participated in 
this study utilized technology in contrast to Grove, Strudler, and Odell’s report that less than one-half of preservice 
teachers used technology in their student teaching experiences. 

The results of the self-reflections indicated all of the preservice teachers used software as production tools 
(e.g., Microsoft Word and PowerPoint) and individualized tracked application (e.g., Milliken Math, Accelerator 
Reader, Cornerstone McGraw Hill Reading, Arthur Readings). Both the preservice teachers and the cooperating 
teachers reported the use of other technologies (e.g.. Leap Pads, PlayStation, overhead projectors, VCR/DVD 
players, tape recorders. 

Smart Boards, laserdiscs, digital cameras and interactive student response clickers). Authoring tools and 
concept mapping were not used by either of the groups of participants. In addition, the Internet was used on a 
regular basis for both teacher and student research and two preservice teachers and cooperating teachers mentioned 
integrating Internet-based digital video and developing WebQuests into the teaching and learning process. 

From their feedback and questions, this was the first time some had really thought about how the use of 
technology was actually implemented: teacher-directed or student-centered. Their understanding of the difference in 
teacher-directed and student-centered use of technology was a bit vague. All five of the preservice teachers felt they 
were using technology in the classroom in both ways. Ayn said, “We make the things we use, but it's the students 
who are interacting with them. I develop manipulatives using the computers, print them out, laminate, and that way 
the students have their hands on activities kind of an indirect way that the students have access to the technology.” 
The involvement she talked about was using the end product (e.g., completing a teacher computer generated 
worksheet). Was this an example of both teacher-directed and student-centered? After some discussion, the group 
agreed perhaps development of the materials and giving them to the students is still teacher-directed. The use of 
technology was teacher- directed, but the activity was student focused. They were trying to sort meanings, within 
their experiences. 

Teacher-directed. According to Wang, (2002), teacher-directed computer usage was defined as teachers 
using computers to: (1) create instructional materials, (2) find resources, (3) communicate with others, (4) keep track 
of students’ grades, (5) present information to the whole class, and (6) provide computer enrichment activity. Six of 
the eight preservice teachers and cooperating teachers recalled using a number of PowerPoint presentations, 
gathering research from the Internet, sharing it with the students, creating manipulatives for student learning and 
creating transparencies to use with the overhead projector. 

Student-centered. In contrast, student-centered computer usage was defined as students using computers to 
(I) create learning resources, (2) find resources for learning activities, (3) communicate with others, (4) present 
information, (5) complete class projects, and (6) engage in more hands-on learning activities (Wang, 2002). 
Therefore, student-centered technology empowers the student. The preservice teachers’ understanding of this 
definition was not always consistent. The K-12 students were actively involved in the use of technology 
independently (e.g., development of crossword puzzles, creating electronic presentations or research on the Internet). 
One of the preservice teachers, Caryn, explained when her children were using the individualized tracking software, 
the students worked independently. These words triggered the student-centered mode for her. “They know how to 
do it. They know exactly what to do, in certain areas.” Gayle mentioned her students used educational software 
called Arthur's Reading during reading time. The software read the words aloud to the children. Some of the 
children used the Paint program, as well. “They created and painted a picture and they enjoyed that.” As the 
preservice teachers continued sharing their thoughts on student-centered use of technology, one “had a child who 
had trouble writing and expressing his thoughts on paper, so when [the class] had our practice writing time, he 
would tape. He would talk and record his story onto the tape.” 

Rae shared her eight graders were not very self-directed because they were used to being told what to do, 
and how to do. They “freeze up” when it is their time. [Preservice teacher and cooperating teacher] have “taken 
baby steps with them and they now know just what to do.” According to the results of the TSA, the preservice 
teachers and the cooperating teachers have a basic working knowledge of computers, software and the Internet. The 
means for the sections Computer Basics, Software Basics and Internet Basics fell within the range of 2 (Somewhat) 
to 3 (Very Easily) in all three areas. 

The section Teacher Readiness to Integrate Technology on the TTQ received the highest means from both 
groups: preservice teachers (M=4.16, SD=.421) and cooperating teachers (M=4.00, SD=.354). The section 
Advanced Skills earned the lowest means in both groups: preservice teachers (M=1.76, DS=.333) and cooperating 
teachers (M=1.42, SD=.375). Therefore, preservice teachers and cooperating teachers felt they were adequately 
prepared and had sufficient support to integrate technology into their teaching using, basic computer, software and 
the Internet, but did not have the more advanced skills (e.g., use advance computer terms publish info in a variety of 
media, determine software-operating system compatibility). 
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The results of the section Using Technology for Learning includes: preservice teachers (M=2.25, SD=.185) 
and the cooperating teachers (M=1.71, SD=.470). This is the only construct with a statistical significant difference. 
The difference in the means of Question 45: “Create and electronic teaching portfolio to evaluate your work.” for the 
preservice teachers (M= 2.38, SD=.518) and cooperating teachers (M=1.25, SD=.463) was of particularly interest. 
The preservice teachers noted this was the first semester they were required to generate both an electronic 
professional portfolio and an electronic student teaching portfolio. 

These results align with other researchers concluding preservice teachers and cooperating teachers are 
using technology in the classroom, to some degree. In most of these classrooms, the use of technology is associated 
with teacher-directed practices and only replaced or improved traditional instruction. Teacher-directed use of 
technology is consider a lower-level use of technology, when it is compared to student-centered use of technology 
(Ertmer, 2005). 

Cooperating Teachers’ Support of Preservice Teachers’ Uses of Technology 

The second theme addresses the effects cooperating teachers have on preservice teachers’ use of 
technology during the student teaching experience. The preservice teachers agreed with Bullock (2004), Grove et 
ai, (2004) and Stevenson (2005). The cooperating teachers do make a real impact upon whether or not technology 
is used for teaching and learning. The preservice teachers acknowledged the support, or lack of support, they felt 
from the cooperating teachers. The relationship was reciprocal, too. Both of these views are discussed below. 

Cooperating Teachers’ Impact. Rae shared her cooperating teacher constantly encouraged her, "Are you 
comfortable with this? Do you feel OK doing this?" There were new things (e.g., using laserdiscs) Rae had not had 
any experience with and she was nervous about using these things with the students. Rae would practice using them 
with her cooperating teacher by her side, until she felt ready to teach using these tools. 

Gayle did not think her cooperating teacher used technology very much, if at all, before she arrived. As 
they talked and worked together, Gayle shared some of her own technology ideas and experiences. The cooperating 
teacher’s response was, "You know, I would like to try that," and she began talking with other teachers in first grade 
about possibilities for integrating technology. The cooperating teacher learned that another first grade teacher had 
created and was using electronic presentations in reading. This news challenged her and encouraged her to begin to 
create presentations on her own. 

Heidi and her cooperating teacher floated between two different schools Teaching English as the Second 
Language. They were working on a new closet-classroom for their students to use in one of the school. Heidi had 
helped her cooperating teacher remove some old computers from the discard pile, with hopes of having more 
computers for the TESL students to use word processing. Heidi’s cooperating teacher indicated these students did 
not have access to computers at home and loved to use the word-processing software. Heidi helped her network 
them to a confiscated printer, as well. Heidi mentioned the paperwork for ESL teachers. There was not a working 
electronic record system in place. She and her cooperating teacher appeared to be very frustrated because the school 
district had purchased them new laptops to transition from the paper record system to an electronic record system 
and they had run into several barriers. 

On the other hand, there were times the cooperating teacher did not seem as supportive and appeared as a 
barrier when the use of technology was the focus. Gayle voiced, “There have been times when I would have liked to 
use an overhead, but my cooperating teacher doesn't have one because she doesn't like to use one. So because of 
that I just write it on the board.” 

One preservice teacher mentioned that her school was filled with technology but she had difficulty in 
finding what was available and where it was hiding. Oftentimes her cooperating teacher would suggest that she try 
specific means to integrate technology (e.g.. United Streaming) and she had no idea they were available. Her 
cooperating teachers shared with her a little at a time. She concluded, “with her not being very excited herself 
about using [technology], she just doesn't think about telling me.” 

Reciprocal Relationships. While the cooperating teacher had a tremendous influence on the preservice 
teachers, the relationship was reciprocal. Echoing the reviews and findings of (Butler & Wiebe, 2003), the 
preservice teachers and the cooperating teachers felt they had exchanged teaching and learning (e.g., using 
voiceover in Microsoft PowerPoint, developing teaching stragtegies using of laptops computers, designing Web 
Quests, working in another platform) with each other. 

The results of the TTQ indicated both groups agreed technology does have a positive impact on classroom 
instruction and student learning. The results of the section Impact on Classroom Instruction included: the preservice 
teachers (M=3.69, SD= .372) and the cooperating teachers (M=3.78, SD=.920) . The results of the section Impact 
on Students included: the preservice teachers (M=3.87, SD= .352) and the cooperating teachers (M=4.2, SD=.754). 
Thus, both groups agree they use technology on a regular basis, the use of technology encourages student-center 
learning and the use of technology improves the quality of student work. 
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In the section Overall Support for Technology in the School the mean is replicated, with the preservice 
teachers (M=3.34, SD= .626) and the cooperating teachers (M=3.34, SD=.823). This constructs included Question 
17 “Teachers in this school are generally supportive of technology integration efforts” and included the support 
given by and perceived by the cooperating teacher. 

Thus, the participants agreed that technology does definitely have a positive impact of classroom 
instruction and students’ learning and teachers within the learning environment support the use of technology. 

Barriers/Enablers Preservice Teachers Face as they Attempt to Use Technology 

The third theme identified common barriers or obstacles and enablers as perceived by the preservice 
teachers. The preservice teachers encountered a number of barriers and enabling factors as they attempted to teach 
using technology during their student teaching experience. 

Barriers 

The results of the interviews support the literature (Collier et ai, 2004; Ertmer, 1999; Evans & Gunter, 
2004; Iding et al., 2002; Persichitte et ai, 2003; Stuhlmann & Taylor, 1999; Williams, 2003) which explored 
intrinsic barriers confronted by preservice teachers who attempt to teach with technology. Gayle voiced it was up to 
her to find ways to use technology. She found herself constantly thinking about how she could integrate technology 
into teaching: “So it's up to me to think!” 

Rae had experienced writing lesson plans with a technology component in her teacher education program. 
She said it looked good on paper. She never attempted to teach one of those plans to “real students. Rae was afraid. 
It was tough!” She shared as she moved into the student teaching with determination and coupled with a 
technology- savvy cooperating teacher, she overcame some of the fear and was successful. 

The participants easily identified extrinsic barriers during their experiences, and some were able to find 
workable solutions. Others reconciled that the students were not missing out on instruction due to a lack of 
resources. Each of these extrinsic barriers, (a) lack of preparation, (b) lack of equipment, (c) reconciliation, (d) lack 
of autonomy, and (e) lack of support are discussed below. 

Lack of preparation. One preservice teacher discussed her prior use of technology in teaching. She did not 
feel comfortable teaching with technology, knowing that so many things could go wrong in the front of thirty 
students and so many computers. She was comfortable with using technology but not teaching with technology. 

Lack of equipment. Three of the eight preservice teachers indicated the computers were “very old and out 
of date” and “we just don't always have the resources to use them in our classroom.” Brooke singled out the fact 
their library of software oftentimes would not run on the classroom computers. Heidi mentioned making sure you 
have all your necessary equipment there: “Eor instance, the other day I had planned to use the overhead and realized 
we didn't have an extension cord that could plug into the wall, and so we had to scrap that and go for working on the 
board.” 

Reconciliation. Two preservice teachers and one of the cooperating teachers indicated the children were 
not missing out on learning. Heidi, elaborating on the nature of Teaching English as the Second Language (TESL), 
felt they received whatever equipment anyone else did not need. She described her classrooms as “large closets” 
with the “left over pieces of disjointed computers” and very few resources. She alluded to the fact she is working on 
a scheduled time for her students to go to the computer lab, “a way to overcome the lack of resources.” 

The Special Education and TESL preservice teachers explained that the structure of the small groups and 
one-on-one teaching were benefits in their respective classrooms, which gave them compensation for the lack of use 
of technology. Ayn concluded, “a computer would be nice, but we really don’t have much room.” A cooperating 
teacher confirmed, “I don't really think the children are losing out on anything.” There again, “they do have the 
computer class once a week, and that at least introduces them to the other aspects of the computer, so they're not 
totally illiterate about it.” 

And yet according to the TTQ, the mean scores for Impact on Students were between (3) Neither Agree or 
Disagree and (4) Agree. The preservice teachers (M=3.82, SD=.352) and the cooperating teachers (M=4.2, 

SD=.754) closely agreed technology does have a positive Impact on Students’ levels of interaction, student learning 
and achievement and has improved the quality of students’ work. So, it appears that the teachers may value the 
impact of technology, but barriers prevent them from making the most effective uses of technology. 

Lack of autonomy. Caryn cited they could only use the good computers in the classroom for Accelerated 
Reader. Heidi, with frustration in her voice, made mention of the fact the school system had just purchased laptops 
for the Teaching English as the Second Language teachers to keep track of the students in different schools. They 
were excited about the laptops, but because of the firewall, teachers were only able to log into their assigned (one) 
school. She and her cooperating teacher served two different schools. “I tried to make it so that we could print 



61 




straight off of the laptop,” Caryn said. “I tried to install the software and couldn't do it.” She indicated she had 
requested help from the school district’s technology coordinator and was waiting for a response. 

Lack of support. The participants aired voices of concern as they talked about specific people who had 
been, or who had not been, identified in their schools as technology savvy. Brooke saw no organized framework 
within the school she could go to for assistance or trouble shooting. She question if there was “a team of people that 
even think about using technology, or are excited about it, or who even know how to do it” and felt very little 
encouragement about using technology just from the people at this school. 

Oddly enough, Rae could identify several sources for assistance. “There is a student that is quite gifted. In 
fact, they pull him out of class on a regular basis to help troubleshoot all over school — sixth, seventh, and eighth 
grade — and he loves it. That's his area, and it's a privilege for him to get to go and do that kind of thing.” And tech- 
savvy teachers are assigned as troubleshooters, in particular areas. 

Cooperating teachers agreed with the voices of their preservice teachers; they did need an easy accessible 
and knowledgeable technology contact person . One spoke of her school having two teachers (e.g., sports coach and 
classroom teacher) assigned as computer assistants. It was difficult to connect with the coach, but the classroom 
teacher was available most of the time. The process for the more difficult troubleshooting entailed a work order sent 
to the central office and a wait time. A similar situation was reflected by another cooperating teacher, but the 
funding for the computer coach was cut. She could not think of anyone else to go to for technology assistance. 

The TTQ indicated the means for the section called Overall Support for Technology in the School were the 
same for the preservice teachers (M=3.34, SD=0.85) and the cooperating teachers (M=3.34 , SD=.823), with (3) 
Neither Agree or Disagree and (4) Agree. The participants agreed to some support for technology in their schools 
was in place. Interestingly, the preservice teachers (M= 3.22, SD=.891) perceived more Technical Support, than did 
the cooperating teachers (M=2.84, SD=.823). To a much stronger degree, the preservice teachers perceived the 
computers were kept in good working condition, they could easily access answers to technology related questions, 
their students have access to updates technology resource and materials and materials (e.g., software, printer 
supplies) for classroom are readily accessible 

Lack of preparation time. The preservice teachers agreed it takes a lot of time and effort up front for a 
teacher to sit down and use technology, Rae said, “It takes forever to create a WebQuest!” Another added using 
technology took up class time. “We have so many skills to teach them. I'd love to teach them how to do 
PowerPoint presentations, because I was so excited when I finally learned how to do it, and how beneficial it is to 
me. There is not time!” 

Enablers 

Once the barriers were set aside, immediately the preservice teachers focused on factors encouraging them 
to use technology for teaching and learning. In many cases these were the barriers in reverse. These included: 
people support and environmental support. 

People support. Rae and her cooperating teacher concluded student feedback would be the strongest 
enabler. “They're excited when we say we are going to go out to the computers today and work. They hate when we 
make them come back in because they're not doing what they're supposed to be.” Caryn, Gayle and Brooke agreed 
that student feedback would be the strongest enabler for them, too. 

Gayle, older that the other preservice teachers, felt she did not have the high school experience in 
technology but “I feel I am prepared to use technology, with my limited background when I came here. I'm not 
afraid to go try to learn something. I might not understand it, but I'm not afraid to try!” In contrast, she shared her 
cooperating teacher was scared to death when she hears computer conference or something like that. “I feel well 
prepared and not afraid at all.” 

Booke mentioned she had an outside source who encouraged her and “got her mind to thinking more about 
using technology.” She shared she felt like the cooperating teacher let them use it “really on my on account that I 
starting doing small groups for technology and a lot of that is my own technology skills.” She says she is 
“proactive.” 

Environmental support. Preservice teachers felt support from websites, easy access to the computers and 
scheduling. 

In two cases, the preservice teachers described the user friendliness (e.g., creation of specific address books 
and distributions lists) of the school’s website. It served as an important tool for communications between the home 
and school. Information was updated often as events evolved in the school’s community, as well as access to email 
accounts. Uploading students’ academic records and grades is forthcoming. 

Rae, her cooperating teacher and Caryn suggested easy access to technology. For them, a number of 
computers were available to them either in the classroom, in the hall adjacent to the classroom or in an easy 
accessible computer lab. The ongoing availability of the computers encouraged the preservice teachers and the 
cooperating teachers to plan for and integrate technology daily. Rae’s cooperating teachers said. “They are just 
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sitting there ready to be used. I just have to signup for them.” Rae felt the longer periods (ninety minutes) of time 
gave the students time to actually “get into the assignment with a technology component. Rae and her cooperating 
teacher agreed their school environment nurtured their technology interest. 

Expectations for the Use of Technology as Perceived by the Preservice Teachers 

The fourth theme described how preservice teachers perceived the expectations for the use of technology as 
required by the teacher education program. Preservice teachers have had similar technology training as required by 
the teacher education program at the university. They have successfully completed a micro-computing course and an 
instructional technology course. The micro-computing course addressed the applications of Microsoft Office and 
Front Page. It is a prerequisite for the instructional technology course. The instructional technology course 
attempted to extend their working knowledge and application of Microsoft Word, PowerPoint, Excel, Front Page, 
and HyperStudio, engaging the preservice teachers in both teacher-directed and student-focused activities to 
integrate into the classroom. This is aligned to meet the current technology standards set by NCATE (National 
Council for Accreditation of Teacher Education, 2003). In an attempt to require preservice teachers to use and to 
build upon their technology skills, there are planned strands of technology integration in the required course work 
throughout the six teacher preparation courses, as preservice teachers developed lesson plans and taught within a 
simulated environment or within a practicum experience. 

The preservice teachers all agreed that they did not feel the use of technology was actually required as a 
part of their student teaching experience and alluded that it was really never a topic of conversation with their 
university’s student teacher supervisor. Ayn, Heidi, Gayle and Caryn agreed technology was not directly discussed 
with the university supervisor, yet felt technology was expected. Ayn implied when she computer-generated 
manipulatives and handouts, the [university student teacher supervisor] always talked about how professional 
looking they were. 

It was not clear as to whether or not the preservice teachers were actually required to have technology as a 
component in the lesson planning and process, during their student teaching practicum. The preservice teachers 
agreed that technology made a positive impact on the classroom and on students’ learning and all attempted to use 
technology on a routine basis, as they had been taught. 

Guidelines for Successful Integration of Technology 

The final theme explores the preservice teachers’ guidelines deemed necessary to be successful in teaching 
with technology in the classroom. As the preservice teachers reflected on their attempts to use technology in the 
classroom, all of them echoed components to a must do list in order to teach with technology, successfully. Their 
lists was based upon their knowledge and their experiences and included (a) allowing for peer interaction and 
collaboration, (b) allowing time for the students to practice technology skills, (c)using technology as another tool 
for teaching and learning, (d) focusing on the classroom management issues, (e) being flexible and (f) understanding 
the use of technology is not for all lessons ,but any lesson that could be enhanced with technology should be (Mims, 
2004). 

The preservice teachers agreed teaching with technology was much more than just learning how to use 
technology. It involved change from traditional teaching. The preservice teachers were not able to readily identify 
the changes but much of their discussion centered on individuals. Interesting enough, an effective use of technology 
integration is student-centered, where students construct their own knowledge (Jonassen, 1991; Nanjappa & Grant, 
2003). 



Implications 

This study was designed to explore preservice teachers’ experiences as they attempted to integrate 
technology during the student teaching experience. The implications of this research are informative to teacher 
educators as they seek to prepare preservice teachers, as well as novice and inservice teachers as they prepare 
themselves to use technology for student-centered learning. 

First, the preservice teacher and the cooperating teachers in this study were using technology on a daily 
basis. They agreed technology made a positive impact students’ learning, and teaching was more student-centered 
when technology was used. Although, their discussion on teacher-directed and student-centered teaching revealed a 
shady understanding of the concepts for preservice teachers, it caused them to reflect. They could easily explain 
teacher-directed learning and connected it with the way they had been taught. It was difficult for them to define and 
give an example of student-centered learning. At a point, the discussion led to the scenario of a teacher making a 
computer generated worksheet and giving it to a student to complete or teaching a student how to use AR 
independently. A broader sense of empowering the students to develop using technology surfaced from the 
discussion. The preservice teachers reflected over their student teaching experiences and identified times they felt 
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they empowered their students to "design and to develop. Their responses included students’ developed WebQuest, 
created PowerPoints with voice-over, and generating a spelling test. 

Prior studies (Beyerbach & Walsh, 2001; Collier et al., 2004; Seels et ai, 2003) noted preservice teachers 
tended to teach as they were taught and oftentimes this was teacher-directed. Preservice teachers must shift their 
thinking of technology integration from thinking they would teach and learn about technology to using technology 
to support student learning. There has been a noticeable shift in preservice teachers’ thought process when teacher 
educators required a technology integration component, scaffold higher levels of technology integration into the 
require coursework, and build on a more focused and comprehensive use of technology (e.g., electronic portfolios) 
(Wright & Wilson, 2005). 

Second, cooperating teachers and the student teaching experience are integral parts of teacher education 
training. In addition, while the cooperating teachers are considered mentors to the preservice teachers, it is evident 
from this study and others (Bullock, 2004; Lane et al, 2003; Margerum-Leys, 2004) that preservice teachers and 
cooperating teachers tap into each others’ curricular, pedagogical and technological knowledge. In this study, the 
preservice teachers perceived the student teaching practicum as being a positive, professional experience, supported 
by the cooperating teachers. 

In this study, when cooperating teachers were actually using technology on regular basic, preservice 
teachers would follow suit, regardless of the level of usage. Early on, it appeared that the preservice teachers were 
trying to duplicate its use, but later in the experiences some of them were trying to think outside the box and carry 
the integration a step farther. Interesting too, when the preservice teachers did not experience tech- savvy 
cooperating teachers, they met the challenged with force and self-motivation and found avenues to integrate 
technology in their teaching. None of the preservice teachers sat idling with technology. 

Preservice teachers were stimulated by attending technology workshops with their cooperating and 
connecting with veteran tech- savvy teachers who were willing to talk infusion of technology experiences. They 
were inspired when they saw computers being lifted from the garbage pile and soon becoming the tool used by 
TESC for expressing the English language. They connected when cooperating teachers brought in their own 
personal laptop for students to experience assistive technology effectively. They connected with cooperating 
teachers who brainstormed troubleshooting, as well as new ideas of technology integration and encouraged them to 
take the next step. They were open to learning new technology capabilities and instructional design, as it was 
scaffold into their learning process, allowing time for practice and the development of its use in their teaching. 

On several occasions, the results from this study revealed the cooperating teachers served as barriers for 
technology integration. The preservice teachers mentioned the cooperating teacher had forgotten to tell her about the 
availability of some technology (e.g.. United Streaming), or the cooperating teachers expected the preservice teacher 
to learn how to use the technology and then teach the cooperating teacher. A preservice teacher reported she was not 
able to use her presentations because the classroom computer would not open them. The cooperating teacher 
encouraged her to use the overhead. In another classroom room, the computer center had been closed for sometime 
and the preservice teacher was told that the students go to the computer lab for their technology. 

As in this study and others (Bullock, 2004; Doering et al., 2003a; Doering et al, 2003b; Lane et al, 2003) 
the cooperating teachers acted at times as an enabler and a disabler as preservice teachers were attempting to teach 
with technology. Obviously, the enabling situations set a positive stage for success. How do the preservice teachers 
respond to these barriers? More work is needed for this areas, even though current studies (Lane et al, 2003; Pierson 
& McLachlan, 2004) suggested the barriers serve as a driving force to push the preservice teacher cope with or 
overcome the proposed barrier established by the cooperating teachers or the preservice teachers are set up as 
change agents within the teaching-learning experience. 

Third, additional barriers and enablers faced by those who attempt to integrate technology are consistent 
with both preservice teachers and cooperating teachers (Brush et al, 2003; Ertmer, 1999, 2003). In this study, 
participants identified both intrinsic barriers (e.g., lack of perseverance, lack of motivation) and extrinsic barriers 
(e.g., lack of time, lack of support). Similar barriers were identified in much earlier studies, as well (Ertmer, 1999) 
This studied indicated there were times the preservice teacher became frustrated, as they faced the barriers not 
knowing exactly what to do. In addition, preservice teachers should be made aware of the potential enablers 
available to them. The participants in this study made use of both people support (e.g., students and cooperating 
teacher) and environment support (e.g., adequate equipment and adequate technical support). Preservice teachers 
should learn how to maximum the power of the enablers, work with the disablers, perhaps even turning them into 
enablers. Bullock (2004) suggested that perhaps the near future would bring about a model for teacher educators 
that encourage preservice teachers to see the factors they encounter as enablers rather than disablers and better 
preparing them as technology using educators. 
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Conclusion 

Preservice teachers are an important means by which technology is introduced into students’ learning 
experience (Evans & Gunter, 2004). Their training throughout the teacher preparation program must align, reflect 
and achieve the most current educational goals, which includes the use of technology for teaching and learning. 

From the results of this study, preservice teachers and the cooperating teachers feel they are adequately trained and 
are capable of integrating technology into their teaching and learning experiences. Both groups are readily using a 
variety of technology in the classroom and are slowly making the transition from teacher-directed uses of 
technology to student-centered uses of technology. The cooperating teachers’ support or lack of support impacts the 
preservice teachers’ uses of technology. Thus, enabling the preservice teachers to tackle barriers (e.g., lack of 
autonomy) and recognize enablers (e.g., the students), as learning takes place for the students, the preservice teacher 
and the cooperating teacher during the culminating student teaching practicum. 
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From a socio-constructivist perspective on learning and instruetion, an exelusive researeh foeus on the effeets as 
sueh of pedagogieal agents on learning ean be questioned. Indeed, learning happens in a soeio-eultural 
environment that as a whole affeets learning proeesses. In order to understand better the impaet of that soeio- 
eultural environment, this study investigates whether eontextual features, or features of the total learning 
environment, affeet the pedagogieal agents' effeet on learning. 

The literature eontains several definitions of pedagogieal agents (Craig & Gholson, 2002; Johnson, 

Shaw & Ganeshan, 1998; Suraweera, 1999). Four main features seem to be reeurrent. First, pedagogieal agents 
are animated eharaeters, i.e. living personas. Seeond, pedagogieal agents have different roles and funetions, all of 
whieh are targeted to support students’ learning. Clarebout, Elen, Johnson, and Shaw (2002) proposed a support- 
typology that ean be used to deseribe these roles and funetions. The typology eonsiders six aspeets of an agent’s 
support: the objeet of support, learner eontrol, adaptability, style, timing, and delivery modalities. Third, 
pedagogieal agents interaet with students: they reaet to students’ aetions. And finally, these lifelike agents are 
part of the digital eomponent of a more eneompassing learning environment. 

Gommer, van Geloven, Jansen, and Zeelenberg (2000) deseribe the total learning environment as the 
totality of means, strategies, persons and faeilities that make learning possible. Students learn through interaetion 
with this learning environment. Traditionally, sueh a learning environment eonsists of the teaeher, peers, 
teaehing materials and other sourees of information, the physieal environment in whieh students are aetive, and 
means sueh as paper, peneils, ete. Aeeording to these authors, the digital eomponent of a learning environment is 
that part of the total learning environment that is supported by information and eommunieation teehnology 
(ICT). Given that pedagogieal agents are part of a digital learning environment, they are at the same time also 
part of the more extensive, total learning environment. Suraweera (1999), for instanee, deseribes pedagogieal 
agents as being part of a digital learning environment, whieh in turn is part of a total, more eneompassing 
learning environment. 

Researeh on pedagogieal agents has reeently attraeted mueh attention (e.g. Baylor & Ebbers, 2003; 
Conati & Zhao, 2004; Clarebout & Elen, 2005, in press; Johnson, Riekel & Eester, 2000; Eester, Converse, 
Kahler, Barlow, Stone & Bhogal, 1997; Eester, Voerman, Towns & Callaway, 1997; Eester, Towns & 

Fitzgerald, 1999; Eester, Zettlemoyer, Gregoire & Bares, 1999; Mayer, Reiser & Eonn, 2001; Moreno, 2004; 
Moreno & Mayer, 2000a; Moreno & Mayer, 2000b; Moreno, Mayer & Eester, 2000; Moreno, Mayer, Spires & 
Eester, 2001; Okonkwo & Vassileva, 2001; Suraweera, 1999; Xiao, Stasko, & Catrambone, 2004). In this 
literature, two types of researeh questions have been addressed. A first question relates to pedagogieal agents' 
impaet on learning results. Researeh on the seeond question aims at identifying whieh of the agent’s features are 
important. These questions, dealt with in the aforementioned studies, typieally address the agent itself within the 
digital environment in whieh it resides. They do not eonsider the total learning environment, or in other words 
the eontext for this digital environment. This is a problem, as the importanee of the total learning environment, 
or the eontext (e.g. the impaet of the teaeher, peers, ete.), ean hardly be overstated from a soeio-eonstruetivist 
point of view on learning and instruetion and with a eoneem for eeologieal valid researeh. 

In this study, it is investigated whether a teaeher's positive, neutral or negative introduetion of a 
pedagogieal agent to the learner affeets the learner’s use and pereeption of the pedagogieal agent, and whether it 
has an (indireet) impaet on learning results. Researeh suggests two eonflieting results of sueh an introduetion. On 
the one hand, in line with the media equation (Reeves & Nass, 1996), whieh postulates that interaetions between 
humans and media are similar to those between humans and even more, “media equal real life” (p.6), one may 
doubt whether the teaeher’s introduetion may ehange the pupil’s opinion and use of the agent as the teaeher is 
simply one other element. Consequently, sueh an introduetion will also have no impaet on the learning results 
attained with the support of the agent. On the other hand, studies in whieh the media equation is eontradieted 
(e.g. Chiasson & Gutwin, 2005) lead to the opposing argument: pedagogieal agents, however humanlike, will 
never be as eredible as a human teaeher. Beeause human teaehers are more eredible, it is most probable that their 
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introduction affects the learner’s use and pereeption of the pedagogieal agent, and even the learner's learning 
results. From this theoretieal assumption an effeet of the teaeher’s introduetion is to be expeeted. More 
speeifieally, a larger, negative impaet of a negative introduetion is to be expeeted than a positive impaet of a 
positive introduetion. This speeifieation is supported by researeh on the so-ealled negativity effect (De Bruin & 
Van Lange, 1999; Fiske, 1980; Hamilton & Zanna, 1972; Peeters & Czapinski, 1990; Reeder & Convert, 1986; 
Riehey, MeClelland & Shimkunas, 1967): “a bias aeeording to whieh evaluatively negative stimuli would have 
greater impaet upon subjeets than equally intense positive stimuli” (Peeters & Czapinski, 1990, p.55). Weinstein 
and Crowdus (1968) give two explanations for the observation that people tend to give more weight to negative 
information than to positive information. First, people expeet normatively ‘good’ aetions, so negative aetions 
attraet more attention beeause they deviate from that norm. Seeond, it is in their own benefit to give more 
attention to negative information: based on this information, people make predietions about the subjeet of the 
information. False-positive predietions are more harmful than false-negative predietions. Riehey et al. (1967) 
give yet another explanation: they state that negative information is seen as more intrinsieally true and honest. 

Assuming the seeond group of researeh results is eorreet, four hypotheses were examined. 

First, it is expeeted that learners who reeeive a positive introduetion of a pedagogieal agent, will listen more to 
what this agent tells than learners who get no introduetion (neutral group). Learners who get a negative 
introduetion of the agent, will listen less to the agent than learners who get no introduetion (neutral group). 
Seeond, it is hypothesized that learners who reeeive a positive introduetion of a pedagogieal agent, will obey this 
agent more than learners who get no introduetion (neutral group). Learners to whom the agent is introdueed 
negatively will obey this agent less than learners who get no introduetion (neutral group). 

Third, learners who reeeive a positive introduetion of a pedagogieal agent are expeeted to pereeive this agent 
more positively than learners who get no introduetion (neutral group). Learners, who get a negative introduetion 
of the agent, are expeeted to pereeive this agent less positively than learners who get no introduetion (neutral 
group). 

Fourth; assuming that being eompliant with the adviee of the agent is funetional towards inereasing learning 
results, learners who reeeive a positive introduetion of a pedagogieal agent, are hypothesized to do better on the 
posttest than learners who get no introduetion (neutral group). Learners who get a negative introduetion of the 
agent, will get lower seores on the posttest than learners who get no introduetion (neutral group). 



Methodology 



Partieipants 

Eighty-one 1 1 years old pupils partieipated. For praetieal reasons partieipants were reeruited in sehools that had 
a eomputer elass at their disposal. In order to reduee elass-effeets eaeh elass was split into at least two groups 
whieh were randomly assigned to different eonditions. Table 1 shows the distribution of the partieipants over the 
three eonditions. 



Table 1: Sample 





POS-group 


NEG-group 


NEU-group 


TOTAE 


Sehool 1 


6 


0 


6 


12 


Sehool 2, elass 1 


0 


7 


7 


14 


Sehool 2, elass 2 


0 


8 


7 


15 


Sehool 3 


10 


0 


10 


20 


Sehool 4 


10 


10 


0 


20 


TOTAL 


26 


25 


30 


81 



Material and instruments 

The digital learning environment 

For the purpose of the study a pedagogieal agent ealled the' Garbage was used. This agent is part of a 

Flemish digital learning environment ealled ‘Afval minderen voor kinderen’ or 'Reducing waste for children' 
(Ovam, 2003), whieh is intended to teaeh 4^^ to graders about sorting waste and reeyeling. In Reducing waste 
for children learners ean visit five different loeations in a virtual sehool (e.g. the elassroom, the refeetory, the 
playground, ete.), in eaeh of whieh they ean explore several objeets and learn about their features (Figure 1). 
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The Garbage Bunny supports the learner throughout the whole program. Using the previously introdueed 
support-typology (Clarebout et ah, 2002), the Garbage Bunny ean be deseribed as follows. First, the Garbage 
Bunny has a eoaehing role. It guides the learner through the program, urges the learner to aet, and provides hints 
for and feedbaek to test items. Seeond, its modality is mainly explaining: the agent explains (parts of) the 
program to the learner. Beeause the Garbage Bunny also asks questions now and then, his modality is 
questioning as well. Third, the objeet of its support ineludes problem solving steps, meta-eognitive aspeets, 
feedbaek during the tests the learner ean perform in the program, and teehnieal issues. The agent offers no 
eontent-speeifie support. Fourth, most of its information is eoneurrently delivered orally and printed. Its delivery 
modality is monologue (not interaetive) and personalized (the learner is addressed in the seeond person). The 
agent also makes gestures to direet the learner's attention. Fifth, learner eontrol over the Garbage Bunny is 
limited as it is always the agent who takes the initiative to provide support, the only eontrol the learner has is to 
listen again to the agenf s explanation. Sixth, support is never differentiated towards different learners. And 
finally, most support is delivered prior to the task, exeept for the feedbaek, whieh is just-in-time. 



Questionnaire. 

To identify whether a partieipant listened and followed the adviee of the pedagogieal agent, a questionnaire was 
eonstrueted. This questionnaire eonsisted of 21 items, i.e. 18 statements (A, B, C, E, F, G, J-U), with whieh a six 
point Likert- type seale (totally agree to totally disagree), and three propositions with a two point seale (agree - 
disagree; D, H and I). Table 2 eomprises the 21 propositions, sorted by hypothesis. For listening and obeying, 
respeetively 4 and 5 independent items were used, so no reliability was measured. The other 12 propositions 
made up one seale to test the pereeption hypothesis. This seale has a strong internal eonsisteney reliability 
(Cronbaeh alpha = .903). 



Table 2: Proposition sorted by hypothesis 



Hypothesis 1 


A. I listened to the Garbage Bunny mostly in the beginning of the session. 


B. I listened to the Garbage Bunny during the whole session. 


C. I listened to the Garbage Bunny mostly at the end of the session. 


D. I listened to the summary of the Garbage Bunny at the end of the test 


Hypothesis 2 


E. I did what the Garbage Bunny told me to do mostly in the beginning of the session. 


F. I did what the Garbage Bunny told me to do during the whole session. 


G. I did what the Garbage Bunny told me to do mostly at the end of the session. 


H. I made use of the underlined words. 


I. I performed a test during the session. 


Hypothesis 3 


J. The Garbage Bunny has helped me. 


K. The Garbage Bunny has a niee voiee. 


L. The Garbage Bunny told stupid things. 


M. The Garbage Bunny looked niee. 


N. The Garbage Bunny has an irritating voiee. 


0. The Garbage Bunny made mistakes. 


P. The Garbage Bunny told niee things. 


Q. The Garbage Bunny looked stupid. 


R. The Garbage Bunny distraeted me. 


S. The Garbage Bunny frequently told the same things. 
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T. The Garbage Bunny did niee things. 




U. The Garbage Bunny was boring. 



Pre-test and post-test 

In order to test learning results, a pre-test and a post- test were eonstrueted. Both tests eonsist of 10 multiple 
ehoiee questions, with five alternatives. The partieipant always got the possibility to ehoose the answer “I don’t 
know” and, by explaining the prineiple of eorreetion-for-guessing, all subjeets were urged to ehoose this answer 
instead of guessing when they would not know the right answer. 

The pre-test assessed partieipants’ prior knowledge. The post- test was a parallel test. In order to avoid problems 
with regard to different levels of prior knowledge, learning gain was assessed and used as the dependent 
variable. In a pilot test, both tests were pre-tested in order to ensure reliability and their parallel nature. In the 
pilot a group of 22 partieipants of the same age answered all test questions. Mean seores were 4.05 (SD=2.13) on 
the pre-test and 4.18 (SD= 1.92) on the post-test. The mean differenee was .14 (SD= 2.03). The paired t-test 
showed no signifieant differenee between the two tests (t(21)=-.315; p>.05 (two-sided)), eonsequently it was 
eoneluded that both tests are suffieiently parallel. The mean seores of both tests also show that they are of a 
reasonable diffieulty. Both pre-test and post-test are internally eonsistent (pre-test KR20= .97; post-test 
KR20=.80). 

Design and Proeedure 

Hypotheses were tested by means of a study with a quasi-experimental pre-test post-test design with three 
eonditions. Partieipants were randomly alloeated to one of the eonditions. 

The pedagogieal agent is positively introdueed to partieipants in the POS-group, negatively to partieipants in the 
NEG-group and neutrally to partieipants in the NEU-group. Four dependent variables were measured: learning 
gain, use of the agent (listening and obeying to the agent) and pereeption of the agent. Figure 2 provides an 
overview of the design 




Figure 2: Design 

Eaeh partieipant followed the same proeedure. First, partieipants were asked to fill in the pre-test. Next, the 
program Reducing waste for children was introdueed. Depending on the partieipant’ s eondition, a speeifie 
introduetion was given. The POS-group was told that the agent is important, that he will help them so that they 
ean aehieve higher seores on the post-test, and that one should listen to the agent very earefully. The NEG-group 
is told the opposite: “although the agent says he’ll help you, he’s rather stupid and repeats himself a lot, so one 
should better not listen to him in order to avoid making mistakes on the post-test or to get too mueh distraeted”. 
To the NEU-group the agent is not explieitly introdueed. 

After the introduetion, eaeh partieipant got half an hour time to work individually with the digital learning 
environment. Thirty minutes is enough to gain suffieient information to eomplete the post- test. To give 
partieipants a notion of what information to look for to sueeessfully eomplete the posttest, they were told that the 
post-test would be about drinking-paekages, so in prineiple they eould find all the information needed in one 
speeifie loeation of the virtual sehool, that is to say the vending maehine. Partieipants were additionally 
motivated by the promise of a reward after sueeessful eompletion of the post-test. 

After the regular working-time or upon request of the partieipant, first the post-test was administered and then 
the questionnaire. 
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Data analysis 

For the dependent variables 'listening', 'obeying', and ‘pereeption', the mean seores were eonsidered. Sinee 
several items were formulated in a negative sense, these seores were first eonverted. 

Item A, B and C, whieh asked if partieipants had listened to the agent, and items E, F and G, whieh asked 
whether partieipants had obeyed the agent, were eaeh eombined into two additional variables respeetively 
‘listening’ and ‘eomplianee’ (respeetively. table 3 and 4). Sinee these variables are eategorieal in nature, Chi 
Square tests were performed to test differenees between the eonditions. 



Table 3: Combination variable ‘Listening’ 



‘T always listened to the 
Garbage Bunny” 


‘T listened to the 
Garbage Bunny mostly 
in the beginning of the 
session” 


‘T listened to the 
Garbage Bunny mostly 
at the end of the 
session” 


Combination Score 
for ‘listening’ 


YES 


YES 


YES 


1 


YES 


YES 


NO 


2 


YES 


NO 


YES 


3 


YES 


NO 


NO 


4 


NO 


NO 


NO 


5 


NO 


NO 


YES 


6 


NO 


YES 


NO 


7 


NO 


YES 


YES 


8 



Tabled: Combination variable ‘Obeying’ 



“I always obeyed to the 
Garbage Bunny” 


“I obeyed to the Garbage 
Bunny mostly in the 
beginning of the session” 


“I obeyed to the Garbage 
Bunny mostly at the end 
of the session” 


Combination score for 
‘obeying’ 


YES 


YES 


YES 


1 


YES 


YES 


NO 


2 


YES 


NO 


YES 


3 


YES 


NO 


NO 


4 


NO 


NO 


NO 


5 


NO 


NO 


YES 


6 


NO 


YES 


NO 


7 


NO 


YES 


YES 


8 



For eaeh of these items, also the single mean seore was analyzed, using ANOVA. The Seheffe-test was used for 
post hoe analyses. For the remaining items (‘T made use of the underlined words”, ‘T performed a test during the 
session” and ‘T listened to the summary of the Garbage Bunny at the end of the test”) the Chi Square Test was 
used. Finally, in order to analyze the partieipants’ pereeption of the Garbage Bunny, whieh is expressed by the 
mean seore of the twelve items regarding the third hypothesis, ANOVA was used. 

To measure the effeet on learning results learners' learning gains were ealeulated by subtraeting their pre-test 
seore from their post-test seore. An ANOVA was performed to study the effeet of eondition with time-on-task as 
a eo- variable. 



Research Results 

Impact of the introduetion on pupil’s use of a pedagogical agent 

Table 5 shows the results of the different analyses for listening to the agent. 
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Table 5: Results for listening' 



TISTENING 








Condition 








Positive intro 


No intro 


Negative 






(n=26) 


(n=30) 


intro(n=25) 


Combination 


1 


23.1% 


16.7% 


0% 




2 


26.9% 


13.3% 


4% 




3 


0% 


0% 


0% 




4 


7.7% 


16.7% 


4% 




5 


11.5% 


16.7% 


56% 




6 


0% 


3.3% 


4% 




7 


23.1% 


30% 


28% 




8 


7.7% 


3.3% 


4% 


Always listen 


Mean 


3.73 


3.40 


1.68 






(SD=1.041) 


(SD=1.632) 


(SD=1.069) 


Mostly in the 


Mean 


4.69 


4.00 


2.84 


beginning 




(SD=1.320) 


(SD=1.438) 


(SD=1.434) 


Mostly at the end 


Mean 


3.04 


2.47 


1.80 






(SD=1.341) 


(SD=1.479) 


(SD=1.190) 


Tistening to summary 


#Yes 


19 


17 


12 


at the end of the test 












#No 


5 


10 


12 



The results for Tistening’ show differenees between the three eonditions. Chi squared analyses show a 
signifieant assoeiation listening and eondition ( % (12)=25.357; p<0.05 (two-sided)). Beeause more eells have 
an expeeted value of less than 5, Fisher Exaet Test was exeeuted. This eonfirmed the findings of the Chi squared 
analyses (Fisher Exaet Test= 24.431, p<0.05 (two-sided)). 

Deseriptives indieate that the three largest differenees between the eonditions are for seores 1, 2 and 5. Almost a 
quarter of partieipants in the positive introduetion eondition says to have listened the whole time, in the 
beginning as well as at the end (seore 1), eompared to none of the partieipants in the negative introduetion 
eondition. While for listening the whole time (seore 4), a differenee ean also be observed between the positive 
and the neutral introduetion eonditions, this differenee is mueh smaller. 

The results for seore 2 (listening all the time but more at the beginning) are analogous. Compared to the neutral 
group, less partieipants in the negative group eondition elaim to have listened to the agent all the time, but more 
in the beginning than at the end. In eontrast, more partieipants in the positive introduetion eondition group than 
partieipants in the neutral eondition state to have listened to the agent all the time, but more in the beginning than 
at the end. 

For seore 5 it ean be seen that partieipants who got a negative introduetion, state far more than members of the 
other two groups never to have listened to the Garbage Bunny. The differenee between the positive introduetion 
eondition and the neutral group is smaller than the differenee between the positive introduetion eondition and the 
negative introduetion eondition. 

All these results are in line with the first hypothesis. 

For the other seores, the differenees between the three eonditions are mueh smaller. While seore 7 (only listening 
at the beginning) appears relatively often in all three eonditions, no partieipant says to have listened the whole 
time but more at the end than in the beginning (seore 3). The number of partieipants falling in the other 
eategories (4, 6 and 8) is rather small in eaeh eondition. 

The three items pertaining to listening were also analyzed separately. The ANOVA shows that the effeet of 
eondition on the three propositions is signifieant (F(2,80)=l 8.410; p<0.05; r|^=0.321 for the first proposition; 
F(2,80)=l 1.378; p<0.05; ri^=0.226 for the seeond; F(2,80)=5.359; p<0.05; ri^=0.121 for the third one). This 
means that eondition aeeounts for 32% in the varianee in “always listening”, for 22.6% in “listening mostly in 
the beginning” and for 12.1% in “listening mostly at the end”. In other words, the effeet of the introduetion is 
strong in the first two eases and moderate in the latter ease (Cohen, 1988). 

Post hoe tests give more information: for the propositions “always listening” and “listening mostly in the 
beginning”, only the negative introduetion group signifieantly differs from the other two groups, while the 
positive and the neutral introduetion group do not signifieantly differ. For the proposition “listening mostly at the 
end”, only the differenee between the positive and the negative group is signifieant. 
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Finally, also a concrete moment of listening to the agent was asked about, by means of the proposition “I have 
listened to the agent’s summary at the end of the test”. No signifieant assoeiation with eondition was found: 

X^(2)= 4.450; p>0.05 (two-sided). 

Table 6 presents the results are presented relating to the eomplianee with the agent. 



Table 6: Results for obedience 



OBEYING 








Condition 








Positive intro 


No intro 


Negative 






(n=26) 


(n=30) 


intro(n=25) 


Combination 


1 


30.8% 


16.7% 


0% 




2 


15.4% 


13.3% 


8% 




3 


3.8% 


3.3% 


0% 




4 


15.4% 


20.0% 


0% 




5 


3.8% 


20.0% 


76% 




6 


7.7% 


3.3% 


8% 




7 


19.2% 


20.0% 


8% 




8 


3.8% 


3.3% 


0% 


Always listen 


Mean 


3.73 


3.60 


1.96 






(SD=1.343) 


(SD=1.653) 


(SD=1.274) 


Mostly in the 


Mean 


4.08 


3.53 


2.24 


beginning 




(SD=1.412) 


(SD=1.676) 


(SD=1.268) 


Mostly at the end 


Mean 


3.42 


2.53 


1.80 






(SD=1.238 


(SD=1.655) 


(SD=1.155) 


Clieking on hyperlinks 


#Yes 


15 


23 


15 




#No 


11 


7 


10 


Exeeuting a test 


#Yes 


24 


27 


24 




#No 


2 


3 


1 



For the effeet of the introduetion on the obedienee to the agent (hypothesis 2), the analysis also started with the 
eombination variable ‘eomplianee’. The Chi squared test shows a signifieant assoeiation between the eonditions 

and eomplianee ( X (14)= 39.584; p<0.05 (two-sided)). Beeause several expeeted values are less than 5, the 
Fisher Exaet Test was exeeuted, whieh eonfirmed the findings of the Chi squared test (Fisher Exaet Test= 

40.136, p<0.05 (two-sided)). Eargest differenees are those for seore 1 (always obeying both at the beginning and 
at the end) and seore 5 (never obeyed). A eomparison of the three eonditions reveals that the positive 
introduetion group is more represented for seore 1 (always obeying, in the beginning as well as at the end). On 
the eontrary, the negative group is far more represented for seore 5, whieh in eontrast to seore 1 represents the 
absenee of eomplianee (never obeying, neither in the beginning, nor at the end). The neutral group falls in 
between the two other groups. All these findings are in line with expeetations and eonfirm the seeond 
hypothesis. 

Also here, separate analyses were done on the individual items. Results of the ANOVA analyses are analogous 
to those for the first hypothesis. For eaeh of the three items, analyses reveal a main effeet of eondition ((F(2,80)= 
11.998; p<0.05; ri^=.235 for the first item; F(2,80)= 10.426; p<0.05; r|^=.211 for the seeond item; F(2,80)= 
8.784; p<0.05; r|^=.184 for the third item). The eondition explains 23.5% of the varianee in “always obeying to 
the agent”, 21.1% of the varianee in “obeying mostly in the beginning” and 18.4% of the varianee in “mostly 
obeying at the end”. In other words, the eondition has a large effeet on always, mostly in the beginning and 
mostly at the end obeying to the agent. But onee again, post hoe tests show that these eonneetions are only 
signifieant between the negative introduetion eondition and the other two eonditions in ease of the first two items 
and only between the negative and the positive introduetion eondition in ease of the third item. 

The two remaining items asked about two eonerete moments of obedienee to the agent: “I have elieked on the 
hyperlinks” and “I have exeeuted a test in the program”. Chi squared analyses showed no signifieant differenees 
between the three eonditions, for neither of the items (respeetively: X^(2)= 2.689; p>0.05 (two-sided) and X^(2)= 
.720; p>0.05 (two-sided)). 
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Impact of the introduction on participants’ perception of the agent. 

Table 7 summarizes the results for the variable ‘perception’. 

The descriptive statistics of general perception (i.e. the mean score for all of the twelve items which asked about 
perception) suggest that the three conditions differ from each other as predicted by the third hypothesis: the 

positive group ( X =4.33, SD=.96) > neutral group ( X =4.29, SD=1.06) > negative group ( X =3.19, SD=.88). 
ANOVA confirms a main effect of condition on general perception (F(2,80)= 1 1.366; p<0.05; r|^=.226). More 
precisely, the nature of the introduction is responsible for 22.6% of the variance in perception. Post hoc analyses 
however show that the difference is only significant between the negative introduction condition and the other 
two conditions. Thus, hypothesis 3, which states that the way the agent is introduced (positively, neutrally or 
negatively) has an impact on perception, is only partly confirmed. 



Table 7: Results for perception of the agent 



PERCEPTION 








Condition 








Positive intro 


No intro (n=30) 


Negative intro 






(n=26) 




(n=25) 


General perception 


Mean 


4.33 


4.29 


3.19 






(SD=0.96) 


(SD=1.06) 


(SD=0.88) 



Impact of introduction on learning gains. 

Table 8 shows the results for learning gains. 

First it was examined if the time the learner worked with the program or the school to which the subject 
belonged had any significant effect on the learning results (e.g. Carroll, 1963). This analysis showed that time- 
on-task had a significant effect on the learning results, (F(l,80)= 6.712; p<0.05; r|^=.082), the school however 
did not (F(4,80)= 1.181; p>0.05; r|^=.059). So it was decided to consider time-on-task as a co-variable in further 
analyses. 



Table 8: Results for learning gains 



EEARNING GAIN 






Conditie 






Positive intro 


No intro (n=30) 


Negative intro 




(n=26) 




(n=25) 


Learning 


Mean 1 .28 


2.27 


2.35 


progression 


(SD=2.15) 


(SD=2.45) 


(SD=3.23) 



The ANCOVA-analyses show no significant connection between condition and learning gain (F(2,80)= ,949; 
p>0.05; r|^=.025). The co-variable ‘time-on- task’ has a significant effect on the learning gain (F(l,80)= 7,553; 
p<0.05; r|^=.091), but because there is no significant effect of condition on time-on-task (F(2,80)= ,942; p>0.05; 
T|^=.024), it seems there is no indirect effect of the condition on the learning results by way of the working time 
either. 



Discussion and conclusion 

This study addresses one main research question: Can features of the total learning environment affect the 
impact of (components of) a digital learning environment? From a socio-constructivist point of view, it is 
important to consider these possible contextual influences. Moreover, this makes this research more ecologically 
valid as in regular settings a digital learning environment is nearly always part of a larger environment. This 
study focused on the impact of one such contextual element, namely the teacher, and on one element of a digital 
learning environment, a so-called pedagogical agent: what effect does the teacher’s introduction have on the use 
(listening and obedience) and perception of the agent by learners and on their learning results? 

While not entirely, results seem to confirm the first two expected effects (use and perception). In case of a 
negative introduction, it was found that pupils tend to listen to and obey the agent less and perceive him less 
positively, than in case of a positive or neutral introduction. However, no effect was found of a positive 
introduction. This finding is clearly in line with research on the so-called negativity effect (De Bruin & Van 
Lange, 1999; Fiske, 1980; Hamilton & Zanna, 1972; Peeters & Czapinski, 1990; Reeder & Convert, 1986; 
Richey et al., 1967). At the same time it raises questions about the limits of the media equation hypothesis or at 
least induces a plea to more systematically consider the context when discussing the impact and effect of digital 
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learning environments. Finally, no effeet of the introduetion was found on the learning results. Beeause an 
existing, eompleted program was used, it was however impossible to turn off the agent. So a first limitation of 
the present study is the faet that there was no ‘no-agent’- eondition. As a eonsequenee, we do not know if the 
Garbage Bunny as sueh has a signifieant impaet on learning results. 

The findings of the present study have several implieations, both for future researeh and for the use of 
pedagogieal agents in sehools. 

First, this study has demonstrated that things told about the pedagogieal agent by the teaeher, and partieularly 
negative information, has an impaet on how mueh students listen to and obey the agent, and how they pereeive 
him. This study gives a first indieation that the total learning environment ean affeet the digital eomponent of the 
learning environment and that interaetions between different eomponents of the total environments and its 
subeomponents may greatly affeet their effeetiveness and funetionality. With respeet to researeh on pedagogieal 
agents, this means that more attention should be given to influenees of the external environment and to the 
eonerete eontexts of use. 

Coneerning the use of agents in elassrooms, these results draw attention to the faet that teaehers should be 
eareful about the way they talk about digital material, e.g. pedagogieal agents. For example, a teaeher who does 
not like pedagogieal agents ean affeet the way the agent is pereeived, by showing his/her opinion to his/her 
pupils. This study however did not eonfirm that this would have a negative effeet on learning results. 

Another limitation of this study relates to the interpretation that parts of the results might be due to a soeial 
desirability bias (Fumham, 1986; Kalton & Sehuman, 1982, Nanearrow & Braee, 2000). The subjeets were 
aware of the researeher’s (fake) opinion about the agent and beeause they were only direetly asked about their 
use and pereeption, it might be possible that they have answered aeeording to what they thought the investigator 
would like to hear. This problem ean be solved in future researeh by using unobtrusive measures, like e.g. log 
files whieh ean register every mouse eliek and by using different researehers to introduee the agent and 
administer the instruments. 

Finally, the plausibility of the theoretieal explanation of the faet that only a negative introduetion has an impaet, 
namely the ‘negativity bias’, remains to be further examined. 
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Patterns in Education: Linking Theory to Practice 
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Joyce Koh 

Abstract 

Analysis of patterns in time and configuration (APT&C) is a measurement and analysis paradigm that 
bridges qualitative and quantitative research methods. APT allows analysis of temporal events (both joint and 
sequential occurrences), whereas APC permits investigation of structural relations or configurations. Temporal 
patterns can be quantified in APT, resulting in probability estimates of their occurrence. Structural relations in APC 
are characterized by properties that are measured quantitatively, such as: passive dependence, independence, 
hierarchical order, complexity, strongness, vulnerability, and wholeness. APT&C is based on theories from 
mathematics and general systems theory. 

Three empirical studies which have used APT&C are described in this article: 1) academic learning time of 
mildly handicapped students in elementary school learning environments; 2) patterns of mode errors in human- 
computer interaction with graphical interfaces in modem software; and 3) stmctural configurations for supporting 
student autonomy in a Montessori classroom. 

The value of APT&C is that results can be directly related to practice. Through APT&C we have new 
ways of conducting educational research in order to shed light on practices that result in the outcomes we seek. 

The Dilemma: Qualitative vs. Quantitative Methodologies 

Research methods in education used for much of the 20^^ century were largely quantitative methods. 
Experimental and quasi-experimental designs were commonplace (e.g., Campbell & Stanley, 1966), and analytical 
techniques included ANOVA, regression analysis and their extensions (i.e., discriminate, factor, canonical and path 
analysis). The basic problem is that this general linear models approach seldom yielded findings that could be directly 
linked to educational practice. Within-group and within-person variance was often large, typically obfuscating 
differences between groups that could be attributed to so-called treatments, practices or programs (Medley, 1977; 1979). 
Cronbach & Snow (1977) further extended ANOVA to deal with aptitude-treatment interactions (ATI), with hopes of 
reducing the within-group variance. But this, too, was seldom successful in yielding significant results. 

In the 1970s and 80s, others began to explore alternative approaches that later became known as qualitative and 
case study methodology (cf Guba & Lincoln, 1985; Stake, 1995; Yin, 2003). Qualitative methods have become widely 
used in educational research in the past two decades. One clear advantage of qualitative methods is that rich details of 
individual cases can give readers helpful insight into and understanding of the educational phenomena investigated. The 
unavoidable dilemma that often accompanies this approach is lack of justification for generalizability of findings. When 
samples are purposive and small, generalizability in the sense of making inferences from sample to population is 
seriously compromised. Indeed, respected books on qualitative methods avoid the term ‘generalizability’ and instead 
employ the notion of ‘transfer’ - i.e., results of what was found in this particular investigation may transfer to other 
similar situations the reader encounters (cf Merriam, 1997). 

Mixed methods approaches have become more popular in recent years (Creswell, 2003), in which both 
strengths of qualitative and quantitative approaches have been utilized. Well before this, an approach that quantified 
qualitative patterns had been proposed: APT. 

Measuring System Dynamics: APT 

Frick (1990) proposed an analytic-measurement procedure called Analysis of Patterns in Time (APT). This is a 
fundamental change in perspective for quantitative methodologists steeped in the linear models tradition and the 
measurement theory it depends on (cf. Kuhn, 1962). The primary difference is that the linear models approach relates 
independent measures through a mathematical function and treats deviation as error variance. On the other hand, APT 
measures a relation directly by counting occurrences of when a temporal pattern is true or false in observational data. 
Linear models relate the measures; APT measures the relation. 
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Academic Learning Time Study 

Frick (1990) conducted a study of 25 systems in central and southern Indiana in which mildly handicapped 
children were observed throughout the day in their elementary school classroom learning environments. Each child was 
observed between 8 and 10 hours across multiple days over a semester. These environments ranged from self-contained 
classrooms for special education students to regular classrooms in which the mildly handicapped children were 
mainstreamed (now called inclusion). Trained classroom observers coded the kinds of academic learning activities 
provided, and within each academic activity the behaviors of target students and instruction made available to the student 
were coded at one-minute intervals. During data analysis, student behaviors at each time sampling point were collapsed 
into two categories: engagement and non-engagement. Similarly, instructional behaviors at each sampling point were 
collapsed into two categories: direct instruction and non-direct instruction. 

Linear models approach. As can be seen in Figure 1, if the data are analyzed with the linear models approach, 
student engagement can be predicted by a regression equation. Approximately 33 percent of the variance in student 
engagement can be accounted for by the amount of direct instruction provided. While this finding shows that there is a 
statistically significant positive relationship {p < 0.05) that is moderate in size, there is still a great deal of uncertainty (67 
percent of the variance is unaccounted for). Notice that the vertical lines (blue) indicate the distances from the data 
points and the regression line (red), indicating errors in prediction. The relationship between direct instruction and 
engagement is represented by a function for a line. In this example, the function for the line is: EN = 0.57 + 0.40DI. 
Each data point represents the overall proportion of engagement for a particular student, paired with the overall 
proportion of direct instruction provided to that student. Engagement is aggregated separately from direct instruction for 
each case, so there is one overall engagement score for a student and one overall direct instruction score. Thus, there are 
25 data pairs from which the regression equation is estimated. 



Linear Regression 

student Engagement = 0,57 + 0.40 * Directinstruction 
R'Sfiuare = 0-33 




Figure 1. Linear models approach to measuring a relation 



APT analysis. The same data were analyzed from an APT perspective. From this perspective data are 
aggregated differently. The joint occurrences of student engagement and instruction were counted in order to form 
probabilities or proportions. For example, for student 1, the probability of (DI & EN) = 0.46; /?(DI & NE) = 0.04; p(EN \ 
DI) = 0.92; and p(EN \ ND) = 0.67. These joint and conditional probability estimates for this student were based on 
nearly 500 data points where the joint occurrences of instruction and engagement were observed and coded. Similar 
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probabilities were estimated for the remaining 24 systems, and then the probabilities were averaged. Thus, there were 
nearly 15,000 data points representing the joint occurrences of direct instruction and engagement in the 25 systems. See 
Table 1. 



Table 1. Temporal Relationships: Joint Occurrences of Direct Instruction (DI), Student Engagement (EN), Non-direct 
Instruction (ND), and Student Non-engagement (NE) in Columns 3-6; Conditional Occurrences in Columns 7-8. 



p(DI) 


p(EN) 


p(DI & EN) 


p(Dl & NE) 


/7(ND & EN) 


/7(ND & NE) 


;?(EN|DI) 


;?(EN|ND) 


0.50 


0.80 


0.46 


0.04 


0.34 


0.16 


0.92 


0.67 


0.39 


0.49 


0.37 


0.02 


0.12 


0.49 


0.95 


0.20 


0.27 


0.56 


0.26 


0.01 


0.30 


0.43 


0.97 


0.41 


0.34 


0.69 


0.34 


0.00 


0.35 


0.31 


1.00 


0.53 


0.48 


0.73 


0.47 


0.01 


0.25 


0.26 


0.98 


0.49 


0.40 


0.75 


0.39 


0.01 


0.35 


0.25 


0.98 


0.59 


0.44 


0.84 


0.40 


0.04 


0.44 


0.11 


0.91 


0.80 


0.36 


0.75 


0.33 


0.03 


0.42 


0.22 


0.92 


0.65 


0.30 


0.67 


0.29 


0.01 


0.39 


0.32 


0.96 


0.55 


0.32 


0.71 


0.31 


0.01 


0.40 


0.29 


0.98 


0.56 


0.42 


0.68 


0.42 


0.00 


0.26 


0.31 


0.99 


0.46 


0.38 


0.84 


0.37 


0.01 


0.47 


0.15 


0.97 


0.75 


0.31 


0.63 


0.31 


0.00 


0.32 


0.37 


1.00 


0.46 


0.54 


0.87 


0.52 


0.02 


0.36 


0.11 


0.97 


0.77 


0.81 


0.92 


0.81 


0.00 


0.11 


0.08 


1.00 


0.57 


0.67 


0.77 


0.62 


0.05 


0.15 


0.18 


0.93 


0.45 


0.24 


0.76 


0.24 


0.00 


0.52 


0.24 


1.00 


0.69 


0.34 


0.74 


0.34 


0.00 


0.40 


0.25 


0.99 


0.61 


0.59 


0.87 


0.58 


0.01 


0.29 


0.12 


0.99 


0.71 


0.52 


0.64 


0.48 


0.04 


0.16 


0.33 


0.93 


0.33 


0.62 


0.83 


0.58 


0.04 


0.25 


0.13 


0.94 


0.66 


0.23 


0.65 


0.22 


0.01 


0.43 


0.34 


0.97 


0.56 


0.29 


0.79 


0.28 


0.01 


0.51 


0.20 


0.97 


0.71 


0.54 


0.75 


0.52 


0.02 


0.23 


0.24 


0.97 


0.49 


0.51 


0.82 


0.50 


0.00 


0.31 


0.18 


0.99 


0.63 




Mean 

(SD) 

0.432 

(0.144) 


Mean 

(SD) 

0.741 

(0.101) 


Mean 

(SD) 

0.416 

(0.139) 


Mean 

(SD) 

0.015 

(0.010) 


Mean 

(SD) 

0.324 

(0.114) 


Mean 

(SD) 

0.243 

(0.104) 


Mean 

(SD) 

0.967 

(0.029) 


Mean 

(SD) 

0.573 

(0.142) 



When direct instruction was occurring, these students were engaged on average about 0.967 (SD = 0.029). In 
the absence of direct instruction, their average engagement was about 0.573 (SD = 0.142). In other words, such students 
were 13 times more likely to be off-task during non-direct instruction [ (1 - 0.573)/(l - 0.967) = 0.427/0.033 = 12.94 ]. 
APT measures the temporal relation between direct instruction and student engagement. The error in prediction is 
indicated by the standard deviation of the mean percent of time. For the /7(EN|DI) mean (0.967), the standard deviation 
was quite small (0.029). In a normal distribution, 95 percent of the scores will fall between 1.96 standard deviations 
above the mean and 1.96 standard deviations below. For the /7(EN|ND) mean (0.573), the standard deviation was much 
larger (0. 142), indicating much more variation in student engagement when non-direct instruction was occurring. In 
other words, student engagement was not only lower but it was less predictable when non-direct instruction occurred, 
compared with occurrences of direct instruction. 

When analyzing the same observational data in APT framework, the conclusions are very clear. For these 
mildly handicapped students, if direct instruction was occurring, then they were very likely to be on-task, and this was 
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very consistent across all students, regardless of the specific learning environment in which they were observed. During 
non-direct instruction, students were much more likely to be off-task, though this was less predictable. These results 
have direct implications for practice. More details are provided in Frick (1990). 

It should be noted that observational data were collected temporally. In this case, joint occurrences of student 
and instructor behaviors were coded at one-minute intervals (time sampling). Alternatively, sequences of behavior or 
events can be observed and coded, as illustrated in the next study, reported below. If observations are not made in these 
ways, it is not possible to conduct APT. As can be seen in Table 1, when independent measures are obtained as would 
normally be the case under traditional views of measurement, one would only have the data in the first two columns for 
analysis. This is how measurement has been traditionally conceived: measure things separately, then relate the 
measures by means of some kind of function. A measure of association is used to indicate the relationship, such as a 
Pearson Product Moment Coefficient. Columns 2-8 in Table 1 were created because an APT perspective was taken in 
the design of how the relation was coded by observers (between student engagement and instruction). One would not 
have the remaining columns of data for analysis had these variables been measured independently. Only because an 
APT perspective was taken was it possible to estimate the conditional probability of the relation, /?(EN|DI), for each 
target system. Moreover, APT measures of relation can be aggregated and subjected to standard methods of analysis, 
such as computation of means and standard deviations, as well as linear regression, illustrated in the academic learning 
time study by Frick (1990). 

Patterns of Mode Errors in Human-Computer Interfaces 

This empirical study by An (2003) illustrates the value of the merging of qualitative methods and quantification 
via APT. She investigated conditions of mode errors when people use modem software. Mode errors occur when the 
same user action results in more than one outcome, depending on the context. Mode errors can cause serious problems 
for software users, such as inadvertent destmction of important work, decreased productivity, and task incompletion. 

Sixteen college students were each asked to perform eight computer tasks during usability tests of three 
modem direct-manipulation software interfaces. Stimulated recall interviews were conducted immediately 
afterwards as subjects watched themselves on videotape to clarify why they took certain actions during the tests. An 
observation system was devised for coding tapes of usability tests to record behavioral patterns of the participants. 

Qualitative analysis of the results indicated three major types of mode errors: A) right action, wrong result; B) it 
isn’t there where I need it; and C) it isn’t there at all. A source of error analysis revealed that mode errors appear to 
result from eight categories of design incongmity: unaffordance, invisibility, misled expectation, unmet expectation, 
mismatched expectation, inconsistency, unmemorability, and over-automation. Consequences of mode errors included: 
can’t find hidden function, can’t find unavailable function, false success, stuck performance, inhibited performance, and 
inefficient performance. 

Analysis of patterns in time (APT) was used as a quantitative method to determine the likelihoods of temporal 
patterns of types, sources and consequences of mode errors. Results of queries regarding temporal patterns were as 
follows: 



Table 2. Sequential patterns of mode errors, their sources, and consequences. 







Query 


Relative 

Frequency 


Likelihood 

(p) 


Type A 


1 


IF type of mode error IS right action, wrong result. 


34 out of 52 


0.65 














a) 


AND IF source of mode error IS unaffordance. 


15 out of 34 


0.44 






THEN consequence IS can ’t find hidden function OR false 
success? 


10 out of 15 


0.67 














b) 


AND IF source of mode error IS invisibility. 


6 out of 34 


0.18 






THEN consequence IS stuck performance? 


5 out of 6 


0.83 














c) 


AND IF source of mode error IS misled expectation. 


7 out of 34 


0.21 






THEN consequence false success? 


6 out of 7 


0.86 
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TypeB 


2 


IF type of mode error IS it isn V there where I need it. 


8 out of 52 


0.15 














a) 


AND IF source of mode error IS mismatched expectation. 


8 out of 8 


1.00 






THEN consequence IS can I find hidden function? 


8 out of 8 


1.00 












TypeC 


3 


IF type of mode error IS it isn V there at all. 


10 out of 52 


0.19 














a) 


AND IF source of mode error IS unmet expectation. 


10 out of 10 


1.00 






THEN consequence IS can I find unavailable function? 


10 out of 10 


1.00 














b) 


AND IF source of mode error IS unaffordance. 


3 out of 10 


0.30 






THEN IF source of mode error IS unmet expectation. 


3 out of 3 


1.00 






THEN consequence IS can Y find unavailable 
function? 


3 out of 3 


1.00 



Query la consists of three phrases. Phrase 1 (IF type of mode error IS it isn ’t there where I need it) occurred 
for 34 out of the 52 mode errors observed {p = 0.65). Given that Phrase 1 was true, Phrase 2 (AND IF source of mode 
error IS unaffordance) was observed to be true for 15 out of those 34 occasions [/7(Phrase 2 | Phrase 1) = 0.44) ]. Given 
that both Phrases 1 and 2 were true, Phrase 3 (THEN consequence IS can Y find hidden function OR false success?) was 
observed to be true for 10 out of those 15 occasions [/?(Phrase 3 | Phrase 2 | Phrase 1) = 0.67) ]. This is interpreted to 
mean that when users did the right action but got the wrong result, the source of user error was often software interface 
elements that lacked affordance (functionality that is not obvious). This frequently resulted in users being unable to find 
a software function that was hidden from view, or they thought they did the task correctly only to find out later they had 
not (false success). When these conditions for modes occurred, software users in this study were unsuccessful 67 
percent of time in tasks they were trying to do. 

While a larger random sample of a broader range of users beyond college undergraduates would be needed to 
increase generalizability, these findings nonetheless illustrate measurement and analysis of sequential patterns of mode 
errors and their estimated likelihoods. These findings offer useful guidance to software designers who want to make 
their products easier to use by minimizing sequences of human errors such as those observed in this empirical study. 

While this particular study does not examine patterns of teaching and learning that would be of more direct 
interest to educators, it does illustrate the value of a difference in approach to measurement. This study demonstrated 
the practical value of mixed methodologies where one first uses qualitative methods for gaining understanding of 
patterns and relationships, and then uses APT to code and quantify those temporal relationships in a manner that is useful 
to practitioners for making decisions based on APT predictions. 

Further information on formal definitions of APT sequences and associated pattern counting rules can be found 
in Frick (1983), Chapters 2 and 5 (note that APT was originally called nonmetric temporal path analysis but the name 
was changed to prevent confusion with conventional path analysis; only the name has changed). 

Measuring System Structure: APT&C 

Thompson (2006) provided the significant insight that APT could be extended to characterize structure or 
configuration of educational systems, in addition to characterizing system dynamics - or processes in education - as APT 
was designed originally to do. Frick and Thompson have since extended APT to measure and analyze configurations in 
education (APT&C, 2005). 

Configural patterns characterize structures in education - i.e., how education is organized, or relations between 
parts. Axiomatic Theories of Intentional Systems (ATIS) provides the theoretical foundation for quantitative measures 
of system structure required by APT&C (Thompson, 2006). These measures include: complexity, hierarchical order, 
heterarchical relatedness, compactness, centrality, flexibility, active dependence, passive dependence, independence, 
interdependence, strongness, unilateralness, weakness, wholeness, and vulnerability. 

ATIS is a systems theory that predicts relationships among system properties, both structural and dynamic. 

There are over 200 axioms and theorems in ATIS. For example, #106 predicts: If system strongness increases, then 
toput increases. See Thompson (2006) and ATP&C (2005) for further details. 
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Koh (2005) was engaged in research on student autonomy. She realized that AXIS properties and APT&C 
would help the analysis of ethnographic classroom observational data she planned to collect. Measuring structural 
properties of systems is a way to characterize classroom structure that supports student autonomy. 

Student Autonomy Structures in a Montessori Classroom 

Autonomy or self-determination was defined by Deci, Vallerand, Pelletier & Ryan (1991) as a state where 
volition for action is totally internalized and determined by intrinsic motivation and not by external conditions. 
According to self-determination theory, intrinsic motivation cannot be fostered if autonomy support was lacking in 
social environments. Experimental studies and self-reported surveys conducted with school-age children found that 
perceived autonomy had a positive impact on perceived competence, intrinsic motivation and conceptual learning of 
school-age children (Grolnick & Ryan, 1987; Valas & Sovik, 1993; Hardre & Reeve, 2003). 

The Montessori system aims to educate each child towards self-mastery and independence (Montessori, 
1964). A distinguishing feature of Montessori classrooms is its provision for student autonomy. In a study 
comparing the social context of Montessori and traditional middle schools, Rathunde & Csikszentmihalyi (2005) 
found that Montessori students reported more support from teachers, more order in the classroom and spent more 
time with academic work rather than in passive listening. 

Koh’s (2005) case study explored how classroom structures support student autonomy in a Montessori 
classroom. Ten one-hour observations were conducted in April, 2006, in an upper elementary Montessori classroom 
located in southern Indiana. It had twenty-eight students, ages 10-12, a Montessori-certified head teacher and two 
assistant teachers. 

Data on interactions between teachers, students and classroom resources were collected through 
ethnographic field-notes. The constant comparative method (Creswell, 1998) was used to identify common 
interaction patterns and classroom activity structures. 

Measures of structural configurations were determined using definitions from AXIS. For example, one 
definition is: 

)/l: Active dependeQt-cotnpoQeut partition measure, 7/i(^S), a measure of initiating 
affect-rebitions. 



=df [di=]....[ndKv) ^ log:|J^J]) ^ Ii] X 100 



An investigator constructs a directed graph representing the affect relations in the system, consisting of vertices (v) 
and edges (e). The degree (d) of a vertex is the number of edges that touch it, and the degree of initiating edges is 
the number whose direction is leading a vertex, (d/vj). \A\ is the cardinality of the set of components in the 
affect relation, where i indexes the different types of affect relations in the system. See reports for further technical 
details on structure measures and AXIS (APT&C, 2005). 

Koh (2005) was interested in the structure of affect relations concerning ‘choice of work’ and ‘guidance of 
learning’. 

Apparent from classroom observations and confirmed through teacher interviews, there were two clearly 
different activity structures in the mornings. Students normally started each day with a new Head Problems 
worksheet created by the teacher, consisting of math and logic-related problems. When this was completed (usually 
within an hour), they typically spent the next three hours in the Morning Work Period, during which students chose 
the type of works they wanted to engage in. Works constituted the major part of their learning goals: research 
projects related to physical science, natural science, history and geography; and book reports, science experiments 
and math workbooks that students needed to complete each nine-week period. 

The Morning Work Period supported student autonomy with respect to which works they wanted to do, and 
whether they wanted to work on them individually or collaboratively. This morning activity pattern of initial Head 
Problems followed by the long Morning Work Period was consistent in all ten observations. 

The data depicted in Figure 2 are from a typical configuration selected from one morning, since this 
configuration was relatively stable from one day to the next. While the specific works chosen by students tended to 
differ each time (as well as the daily worksheets for Head Problems), the activity structures with respect to the 
‘choice’ and ‘guidance’ affect relations were highly similar (homeomorphic). 
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‘Centrality’ measures the number of indirect connections from each primary initiating component (i.e., one 
with a directed edge from that component which does not have a directed edge to it) to all others. 

During Head Problems, ‘centrality’ was found to be substantially higher since the teacher chose the same activity 
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Figure 2. Measures of structure of affect relations for ‘choice’ and ‘guidance of learning’ 



for all students to work on. In comparison, during the Morning Work Period the students chose 
various specific works they wanted to engage in (from literally hundreds of resources available) and who they 
wanted to work with. Correspondingly, ‘complexity’ (the number of connections between teachers, students and 
resources (works) in the classroom) was also higher during this time, since nearly every student was typically 
engaged with a different specific work (i.e., individualized instruction via engagement in that work). 

‘Active dependence’ measures the number of emanating paths where connections were initiated, while 
‘complete connectivity’ measures the structure where connections were both initiated and received. During Head 
Problems, ‘active dependence’ was higher since the teacher chose the head problems for all the students to work on. 
Without corresponding choice from students, ‘complete connectivity’ was non-existent. During ihQ Morning Work 
Period, ‘complete connectivity’ characterized the structure regarding each student who selected the particular work 
which in turn guided their learning. 

‘Interdependence’ was higher during the Morning Work Period as there were more instances of children 
choosing to work collaboratively with peers. Consequently, ‘independence’ was lower, as there were few instances 
of primary initiating components with respect to choice and guidance. Observations and interviews with teachers 
also indicated that free-flowing nature of the Morning Work Period enabled them to have one-to-one feedback 
sessions with students on report drafts written about their individual works. These sessions could be as long as 45 
minutes per student and gave teachers the opportunity to personalize instruction and correct errors. 

The three teachers’ responses to the Problems in Schools Questionnaire (SDT Website, 2006a) showed all 
three of them to be ‘Highly Autonomy Supportive’. They valued encouragement, empathy and student viewpoints 
over the use of extrinsic rewards and punishment. This was evidenced by them encouraging students to be critical 
about the Head Problems. When there was missing or wrong information that might hinder problem solution, 
students were encouraged to provide suggestions and help contribute to the problem solution by researching for the 
required information. 

Nevertheless, support for autonomy did not preclude the need for classroom management. Teachers were 
observed to be unhesitant to manage students when there were disciplinary problems or who were off-task during 
both the Morning Word Periods and Head Problems. This corresponded with Montessori’s philosophy that while 
student choice is respected, students who disrupt learning are stopped and redirected. 

The structural configurations and teaching strategies were found to have a positive impact on the extent to 
which students felt intrinsically motivated to learn. Student responses to the Academic Self-Regulation 
Questionnaire (SDT Website, 2006b) indicated that they had a greater tendency to undertake learning activities 
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because they perceived some personal value and identification with the learning goals, rather than because they felt 
compelled by external factors. 

It should be noted that Koh (2005) was able to construct digraphs of relations between components in the 
classroom, i.e., between specific students, their works, and their teachers. Those digraphs do not depict temporal 
interactions. Rather they indicate a set of structural relationships, such as student Mary chose a research activity in 
cultural geography in a subtropical climate (this is an affect relation concerning student choice of work). And, in 
turn, Mary’s learning was guided by the instructional materials on biomes and card materials on animals and foods 
in that biome (this affect relation concerns the guidance of learning). In a similar manner, the remainder of the 
digraph was constructed for other students and works chosen, including a few students who were receiving 
individual guidance from their teachers. 

While the specific works chosen would change from day to day, and even within the morning work period, 
the structural pattern of relationships was relatively constant. Mathematically speaking, the structures were 
homeomorphic. The constructed affect relation matrix was then entered into a software prototype for doing APC 
property measure calculations, and those quantitative results were reported in Figure 1. Further information on 
structural measures such as centrality and independence can be found in report 1 1 , ATIS Graph Theory, by Kenneth 
Thompson (2006) at: http://education.indiana.edu/-aptfrick/reports/llATISgraphtheory.pdf . 

Can I Do APT&C with my Existing Data? 

Ordinarily, the answer to this question is no. One must observe and record the temporal patterns before 
they can be counted. Configurations must also be observed and recorded in order to measure structural properties. 
Digraphs or corresponding matrices can be constructed that indicate the structure of the observed system. 

While it is entirely possible to do APT&C by hand with the aid of a pocket calculator, it can become 
extremely tedious to do the counting, as well as error prone. Computer software to assist in both data collection and 
analysis is under development at Indiana University. For further information, see the APT&C Website at: 
http ://www. indiana. edu/-aptfrick . 

Summary 

Through APT&C we have new ways of conducting educational research in order to shed light on practices 
that result in the outcomes we seek. APT is a way to measure and analyze temporal patterns, including system 
dynamics and processes. APC is a way to measure and analyze configurations of structural relations in education. 
APT&C is a paradigm shift in how we observe and measure phenomena in education and other settings. APT&C 
contains both qualitative and quantitative aspects, yet it is neither quantitative research methodology as traditionally 
conceived, nor is it qualitative methodology. APT&C is grounded in mathematical theories (set theory, probability 
theory, graph theory, topological theory) and general systems theory. 

In this paper, three different research studies were summarized. One focused on mildly handicapped 
student and instructor behavior in elementary school settings; another looked at adults trying to use modem 
computer interfaces and their stmggles due to mode errors in the design of these software products; the third 
investigated how student autonomy is supported by stmctures of affect relations in a Montessori elementary 
classroom. In each of these analyses, the results are clearly applicable to practice. Hence, APT&C is a research 
methodology that helps link theory to practice. 
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Abstract 

In this study the author investigated whether a relationship exists between teachers’ learning styles and their 
self-reported technology related needs, beliefs, stages of adoption, software expertise, and technology competencies. 
Learning style was identified using the Gregorc Style Delineator. The Snapshot Survey measured technology related 
needs, beliefs, stages of adoption, and software expertise, and technology competencies were measured using the 
Technology in Education Competency Survey. The final analysis included data collected from a sample of 499 
participants from 12 elementary schools. 

The data suggest a significant relationship between learning style and the technology related needs, stages of 
adoption, software expertise, and competencies of teachers. The relationship between learning style and technology 
related needs was significant at the p < .01 level, and between learning style and technology related stages of 
adoption, software expertise, and technology competencies at the p < .05. 

Possible applications of these findings may include mentor programs and training models that have been 
individualized to match learning style profiles. 



Introduction 

Since the beginning of the Industrial Revolution, successive waves of new and domestically developed 
technologies have fueled economic growth in the United States. Where foreign competition has eroded American 
predominance in a given field, the dramatic growth of another sector has provided jobs to replace a share of those 
that have been lost. 

The last thirty years of the twentieth century, when many multinational corporations exported factories and 
jobs to the developing world, provide a key example of this phenomenon. Innovations in computer hardware and 
software production led to the development of new industries in which a comparatively highly trained, expensive 
workforce was an asset. Innovators such as Steve Jobs and Bill Gates have played critical roles in the creation of 
jobs employing millions of American workers. 

Now, at the end of the first five years of the new millennium, foreign competition again threatens to siphon 
jobs away from the United States. According to the American Electronics Association (AEA), “High-tech 
employment totaled 6 million in 2002, down from 6.5 million in 2001” (p.l). This may be true because a dramatic 
decrease in the cost of telecommunications and the growth in the number of qualified overseas workers have made it 
possible for American businesses to develop software more economically and provide technology related support 
services from foreign countries (Information Technology Association of America [ITAA], 2003.). The ITAA’s 2003 
survey of 374,129 companies found that twelve percent of companies engaged in IT commerce outsourced jobs 
overseas. The types of jobs most likely to be sent overseas require more skill and command higher wages than might 
be expected: 

While offshore development is often considered an alternative for low end 
“commodity” work, the ITAA survey finds programming/software engineering 
the job category most likely to go overseas (67%), followed by network 
design (37%) and web development (30%). In terms of jobs going overseas 
in the next six months, companies cited programming/software development (35%) 
and technical support (29%). (ITAA, 2003, p. 12) 

This state of affairs may leave one asking about the source of America’s next great economic engine. Perhaps 
because the next generation of innovators and entrepreneurs is currently enrolled in K-12 schools across the country, 
concern has grown about the need to prepare these students for the competitive environment that they will face. 
Efforts to provide technology tools and training to them are substantiated, for example, by data from such as the 
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National Center for Educational Statistics [NCES], which estimates that American schools spend over $5 billion on 
educational technology (2003). 

In some respects, these efforts have been successful. Computers and Internet access currently are much more 
widely available in schools. Since 1983, the total number of computers in U. S. classrooms is estimated to have 
grown from less than 50,000 to more than 5.8 million (Mehlinger, 1995; U. S. Congress, Office of Technology 
Assessment, 1995). Between 1998 and 2000, the percentage of U. S. classrooms connected to the Internet rose from 
27% to 63% (Revenaugh, 2000). Between 1996 and 2000, over I million classrooms were wired to the Internet 
(Carvin, 2000; U. S. Congress, Office of Technology Assessment, 1995 as cited in Rovai and Childress, 2002). The 
growth in federal spending on technology from $21 million in 1995 to $729 million in 2001 has also helped to 
accelerate access to computers in schools (Russell et al, 2003). 

Yet research suggests that student use of computers has not increased. A 1997 study of teachers and 
administrators sponsored by Jostens Learning Corporation found that teachers were more likely to create documents, 
spreadsheets, or graphics for “personal productivity” than for instruction (1997, as cited in Earle, 2002). In a recent 
telephone survey, 39% of middle and high school students said that they “hardly ever” use computers in school 
(Public Agenda, 2001). The Snapshot Survey, conducted with 3,665 teachers in four states, found that 14% of K-12 
teachers do not use technology for instruction, and 45% use it for instructional purposes fewer than fifteen minutes 
per week (Norris et al, 2003). 

If there are more computers in classrooms and Internet access is more widely available, why are students not 
using computers for instructional purposes at a higher rate? If time is not being allocated for the use of instructional 
technology, it seems advisable to investigate relationships between the skills and beliefs of the people in public 
schools who are responsible for scheduling the majority of the school day. In self-contained and semi- 
departmentalized elementary settings, teachers perform this function. As a result, it is possible that the use of 
educational technology in K-12 environments may be limited by the degree to which classroom teachers embrace it. 
Teachers may not only avoid using technology for instructional purposes themselves, but also fail to provide 
opportunities for their students to do so. 

So the question becomes, do the learning styles of teachers have a relationship with the degree of comfort and 
proficiency with which they work with educational technology? Specifically, do the learning styles of teachers relate 
to the technology related needs, beliefs, levels of adoption, software expertise, and competence related to 
educational technology? This study was completed to determine whether teachers with the same dominant learning 
style share similar characteristics with respect to their beliefs about, and proficiency with, educational technology. 

Literature Review 

The study of learning styles, their interaction with technology use patterns, and their impact on a variety of 
other technology-related characteristics have generated a considerable volume of research. A significant body of this 
research explores factors that determine if technology is used in the classroom for instruction. Many of these factors 
identify the personal characteristics of teachers as possible obstacles to using computers for instructional purposes 
(Christensen, 2002; Milbrath & Kenzie, 2000; Ropp, 1999). 

In a quantitative study of 94 classrooms, Baylor and Ritchie (2002) investigated the impact of the “planning, 
leadership, curriculum alignment, professional development, technology use, teacher openness to change, and 
teacher non-school computer use” on student learning and teacher skill and morale (p. 395). Stepwise regression was 
used to determine which combinations of variables affected one another. The study found that teachers’ openness to 
change and use of technology had an impact on the degree to which those teachers used technology in the classroom 
for instructional purposes. 

Citing the work of other researchers, Earle (2002) identified access, time, training, cultural acceptance, 
leadership, and technical support as factors that needed to be present to successfully integrate technology into 
instruction. Knezek et al (2000) used structural equations to formulate and investigate the validity of the Will-Skill- 
Tool model. They found that teachers who have the will to use instructional technology, the necessary skills, and 
access to the appropriate hardware and software tools are more likely to use technology in the classroom than 
teachers who do not. 

Becker’s analysis of the Teaching, Learning, and Computing Survey — a study of over 4,000 teachers of 
grades four through twelve in over 1,000 American schools — indicated that teachers rated in the top 25% with 
regard to computer skills, variety of use of computers, and experience working with computers “had students use 
three times the number of types of software than teachers in the bottom 25% (2000, p. 12). 

Dursick (1998) found that “personal social cognitive factors” such as attitude, anxiety, self efficacy, 
willingness to make time commitment, [willingness to] face risks, competency, beliefs, perceptions of relevance. 
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and lack of knowledge” have an impact on technology use in the classroom. George and Camarata (1996) attribute 
the reluctance of teachers to use technology to a lack of confidence in their abilities to do so. 

Even though teacher characteristics have been widely identified as factors that have an impact on technology 
use in schools, the significance of such factors is not universally accepted. Using Snapshot Survey data from 3,665 
teachers in California, Florida, Nebraska, and New York, Norris et al (2003) found that technology use was almost 
exclusively a function of access, to the exclusion of the individual differences of teachers. Conlon and Simpson 
(2003) have also found that individual differences among teachers, with respect to computers, appears to be fading 
as more computers are found, and frequently used, in teachers’ homes in Scotland. 

As a result of this ongoing dispute, research on beliefs and personality characteristics is arguably a more 
pressing need than other factors such as access and staff development. This is because the personal characteristics of 
teachers may be the more challenging obstacles to address. Instituting additional staff development programs and 
purchasing more equipment may remedy skill deficits and increase the number of computers available for teachers 
and students to access. To create high quality staff development programs that promote the use of instructional 
technology, personal attributes of teachers need to be identified and addressed. Teachers who are forced to 
experiment independently with new equipment or participate in workshops that do not meet their needs may develop 
more resistance to technology. People who have experienced frustration in learning to use technology are more 
likely to be resistant to it or have anxious feelings about it, depending on the reasons that they believe they have 
failed (George & Camarata, 1996; Martinko, Henry, and Zmud, 1996; Morgan, Morgan, & Hall, 2000). These 
findings suggest that if an individual participates in training in a manner than does not result in success, initial 
resistance to technology may be further entrenched. 

Cognitive Styles Research 

Research in the area of cognitive styles is focused on identifying factors that distinguish the thought processes 
of one person from those of another. It helps to explain why individuals think and act in different ways when 
presented with the same types of sensory stimuli. Work in this area began in the late 1800s when psychologists 
attempted to identify a link between low level, observable cognitive processes such as sensory acuity, finger tapping, 
and speed reacting to stimuli (Dillon and Watson, 1996). 

Early efforts in cognitive styles research sought to explain the role of personality in determining how people 
think and act. Karl Jung, one of the earliest researchers to explore this field, believed that people learned in different 
ways partly because of their unique personalities. Jung’s model of personality types laid the foundation for the 
Myers-Briggs Type Inventory that is used today (1923, cited by Sternberg and Grigorenko, 1997). 

There is an immense amount of material that has been published over the past sixty years related to the 
concept of cognitive styles. Messick’s model identified nine different styles, including scanning, leveling- 
sharpening, constructed- flexible control, tolerance for incongruous or unrealistic experience, field-dependence, 
cognitive complexity, reflection-impulsivity, styles of categorization, and styles of conceptualization (Messick 1970, 
cited in Goldstein and Blackman, 1978). By 1976, his list had grown to nineteen. It is challenging to produce a 
comprehensive overview of the literature because a complete, coherent, and widely accepted framework does not 
exist for categorizing the multitude of constructs into a single system. Sternberg notes that this has created problems 
for researchers in the field: 

...the styles literature has failed to provide any common conceptual framework 
language for researchers to communicate either with each other or with 
psychologists at large... The result is a kind of balkanization of research 
groups, and balkanization always has led to division and, arguably, deaths 
of a thousand cuts (2000, p. 250). 

The intensity of interest and research activity in the area also appears to be cyclical. In 1978, Goldstein and 
Blackman cited this interest as a reason for writing their book. Almost twenty years later, Sternberg and Grigorenko 
speculated that another active era in cognitive styles research may be at hand, musing that cognitive styles, “much 
like wide neckties,” seem to go in and out of fashion on a cyclical basis (1997, p. 701). This further complicates the 
task of encapsulating all of the relevant research into one review. 

Sternberg and Grigorenko (1997) have developed a list of common characteristics that they assert are shared 
by all valid models of cognitive style (Table 1). They also distinguish between cognition-centered, personality- 
centered, and activity-centered approaches to cognitive style research (1997). This taxonomy provides a framework 
for reviewing a diverse field of study, and it has been repeatedly cited in the literature (Cano-Garcia & Hughes, 

2000; Rayner & Riding, 1997; Zhang, 2002a; Zhang, 2002b). 
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Table 1 

Characteristics of Valid Cognitive Styles Models 



Characteristic Description 

Theoretical Specification The positing of a reasonably complete, well- 

specified, and internally consistent theory of 
style that makes connection with eirtant 
p sy c ho 1 0 gic al the or y . 

Internal Validity A demonstration by factor analysis or some 

other method of internal analysis that the 
underlying structure of the item or subtest 
data IS predicted by theory. 

Convergent External Validity A demonstration that the measures of styles 

correlate with other measures with which, in 
theory, they should correlate. 

Discriminant External Validity A demonstration that the measures of styles 

do not correlate with other measures with 
which, in theory, they should not correlate. 

Heuristic Generativity The extent to which the theory has spawned 

and continues to spawn psychological 
research and, ideally, practical application. 

Note. Adapted from “Are cognitive styles still in style?” by Robert J. Sternberg and 

Elena L. Grigorenko, 1997, Amsri can Psychologist, 52(7), p. 703. Copyright 1997 by the 

American P svcholo sic al Association. 




Cognitive-Centered Approaches 

The field-dependence/field-independence model is an example of a cognitive-centered approach to cognitive 
styles research whose roots are in the field of psychology. Hayes and Allinson describe this model as “the most 
thoroughly researched dimension of cognitive style” (1996, p. 67). The researchers explored how subjects were able 
to determine their physical orientation to the ground without any visual cues (Witkin, Moore, Goodenough & Cox, 
1977). They discovered that some people rely on perceptions of the area around them, or the prevailing field, for 
cues about the structure of their environment. This mode of perception is known as field-dependence. The ability to 
perceive without cues from the surrounding environment is known as field-independent. 

The educational implications of this model are that people with a field-independent learning style should be 
able to learn with fewer contextual cues, while field-dependent learners will be more likely to rely on them. Witkin 
et al (1977) was among the first cognitive styles researchers to suggest that teaching and learning styles could be 
matched to improve student outcomes. 

Hayes and Allinson (1996) found that two of six studies using the field-dependence/independence learning 
style model to investigate the possible effect of matching learning styles on learning outcomes demonstrated a 
significant relationship. Smith believes the field dependence/independence model favors males from middle and 
upper class backgrounds (2002). He also argues that the use of Witkin’ s work to formulate spatial components of IQ 
tests illustrates how being field independent is perceived as being better than field dependent, because people who 
demonstrated field independent qualities on an IQ test would earn higher scores (2002). 

Activity-Centered Approaches 

Although the terms cognitive style and learning style are often considered interchangeable, Sternberg and 
Grigorenko consider learning style a subset of cognitive style (1997). This designation is based on the assumption 
that learning is an activity, an assertion made by Kolb in his Experiential Learning Model. According to Kolb, the 
learning process consists of “concrete experience, reflective observation, abstract conceptualization, and active 
experimentation” (Boyatzis & Kolb, 1991, p. 279). The Kolb Learning Styles Inventory [LSI] measures the reliance 
of an individual on each of the four stages (Takacs, Reed & Wells, 1999). The two most prominent stages for each 
individual combine to define his or her learning style. Under this model, one may be a di verger, assimilator, 
converger, or accommodator (Cano-Garcie & Hewitt-Hughes, 2000; Terrell, 2002). Divergers prefer concrete 
experiences; assimilators “create models for tasks at hand;” convergers conceptualize abstractly and experiment 
actively; and accommodators learn through processes (Takacs et al, 1999, p. 2). 

The Kolb Learning Styles Inventory also has been applied in research related to computer users. In an analysis 
of five studies completed before 1990, Hayes and Allinson (1996) found that all five indicated that there was some 
interaction between learning style and achievement. Wang, Hinn, and Kanfer (2001) found no significant difference 
between Kolb learning style groups on an achievement test taken by 3 1 participants following a class that used a 
computer program to facilitate online collaboration. 

Terrell found that there was a relationship between learning style and completion rate for a sample of 159 
doctoral students (2002). Using ANOVA and independent sample t-tests, Terrell cautions that the Kolb LSI has been 
widely criticized for construct validity (2002). The activity-centered model proposed by Dunn and Dunn is based on 
the active nature of learning. This model is oriented to the way in which an individual reacts to environmental 
factors related to the acquisition of new information (Howard, 1998; Kaplan & Kies, 1995; Sims & Sims, 1995). 

Raupers (1999) found that teachers performed better on a posttest following a six-hour technology workshop 
when exposed to a culminating activity designed for their unique learning styles, as assessed using the Productivity 
Environmental Preference Survey [PEPS] (Dunn, Dunn & Price, 1991; Raupers, 1999). This study suggests that 
technology classes structured to accommodate different learning styles may better meet the needs of adult learners. 
Sternberg and Grigorenko observe that the Dunn and Dunn model refers “more to elements that affect a person’s 
ability to learn than to ways of learning” (2000, in Sternberg & Zhang, p. 17), such as the temperature of the room 
and the emotional state of the participants. 

Personality-Centered Approaches 

Personality-centered approaches include those that, although they may influence or impact cognitive 
processes, are rooted in personality traits (Sternberg and Grigorenko, 1997). The Myers-Briggs Type Indicator 
[MBTI] is one example of a personality-centered model that is derived from Jung’s personality types (Garland, 
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1990; Myers & McCaulley, 1989). Jung’s eight types are based on four individual subscales: introversion and 
extraversion; sensing and intuitive; thinking and feeling; and judging and perceiving (Catland, 1990). Moreland 
(1990) asserted that individuals might struggle with technology because of their personality types. 

Smith and Munday (1995) found that intuitive and thinking types of teachers were more willing to use 
technology than sensory types who were considered more practical in their orientation. Sensory/feeling types were 
least likely to be comfortable with technology. Chambers (et al, 2003) used the MBTI and the questionnaire 
developed by Smith and Munday (1995) to find that intuitive/thinking types are most likely to be comfortable with 
technology, while sensory/feeling types were least likely to be comfortable ( Chambers et al, 2003). 

The Gregorc Style Delineator (GSD), also a personality-centered approach to cognitive styles research 
(Sternberg & Grigorenko, 1997), includes elements of personality and cognition as reflected in behavior. Gregorc 
explains that “[IJeaming style consists of distinctive behaviors which serve as indicators of how a person learns from 
and adapts to his environment” (1979, p. 234). The combination of attributes that enables one to use a given set of 
channels to make sense of one’s environment is called mediation ability, and the behaviors that result from the use 
of this ability are known collectively as style (Gregorc, 1982a, p. 5). The model classifies individuals with respect to 
how they receive information, which is perception, and the way in which they organize it, which is ordering 
(Gregorc, 1982a, p. 5). The synthesis of the two dominant qualities within the perceptual and ordering domains 
results in the four channels in the Gregorc model. These include concrete sequential [CS], abstract sequential [AS], 
abstract random [AR], and concrete random [CR] (Gregorc, 1982b). 

CS people thrive on structure. They are most comfortable learning skills and completing tasks that are 
accompanied by step-by-step directions and clear criteria for what is considered mastery or successful completion. 
Possible positive characteristics include good organization skills and proficiency with handling details. Possible 
shortcomings of the CS mind style include rigidity and a heavy-handed leadership style characterized by a tendency 
to micromanage (Gregorc, 1997). 

People with the AS mind style are considered more academic, philosophical, and analytical in their 
orientation. They are thoughtful people who ask questions; they prefer to know why they have been instructed to 
learn a skill or complete a task as well as how they are supposed to do it. Strengths of AS people may include a deep 
understanding of their chosen fields and strong analytical skills. Shortcomings may include the perception among 
others that they have a tendency to fixate on and repeatedly challenge small parts of ideas not perceived to be 
compatible with their own (Gregorc, 1997). 

The AR mind style is closely linked with human emotion. AR people learn skills and perform tasks more 
readily when they relate well to their teachers or their supervisors. Strengths of ARs may include a gift for 
interpersonal relations, artistic creativity, and an ability to interpret the feelings of others. The disadvantages include 
a lack of organization skills, mood swings, and a poorly developed concept of time (Gregorc, 1997). 

CR people are likely to be competitive and intuitive innovators. They typically do not enjoy working with 
details or following routines, and they prefer to learn skills and complete tasks independently. Strengths of CRs may 
include possessing the attributes of visionary and driven leaders. Their weaknesses may include being overly 
competitive or taking excessive personal or professional risks (Gregorc, 1997). 

A variety of studies have used the Gregorc mind styles model to investigate the relationship between learning 
style and technology-related attributes. Ross and Shulz (1999) found that learning style did not have a significant 
impact on learning in a computer assisted instruction [CAI] environment. Ester (1994) found that abstract learners 
performed significantly better using a lecture approach than they did using a CAI system, but he did not find a 
significant difference between the achievements of people with different learning styles when the same instructional 
methods were used. 

Ames (2003) found a relationship between learning style and computer attitudes using a shortened form of the 
Computer Attitude Scale (CAS) and the GSD. The study found that AS people were “more confident, less anxious, 
and more favorably disposed to instruction via computer,” while AR and CR learners were “less inclined to be 
receptive to technology-facilitated instruction” (Ames, 2003, p. 10). Because women in the sample population were 
more likely to be AR learners, Ames also called for more research into a possible relationship between learning 
styles and attitudes toward technology. 

One consistent finding among these studies is that AR learners do not appear to function as well in 
environments that involve computer equipment, or learn as well in classes that address computer topics, as other 
types of learners. This, in turn, suggests that there may be a relationship between the technology-related attributes of 
teachers and their learning styles. 

The literature review illustrates that a relationship may exist between cognitive style and several key 
technology-related attributes among teachers. Due to the small number of studies that have made use of the GSD, 
research has been drawn from cognitive-centered, activity-centered, and personality-centered traditions within the 
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field of cognitive styles research, as identified by Sternberg and Grigorenko (1997). Findings are not universally 
consistent, but they do provide reason to believe that people with different learning styles may react differently to 
training based on their styles (Hayes & Allinson, 1997). 

No prior studies have been undertaken with elementary school teachers using Gregorc’s original 
methodology with this combination of variables — technology-related needs, beliefs, stages of adoption, software 
expertise, and technology competencies. The use of an elementary teacher sample is important, based on the 
findings by Norris et al (2003) that secondary teachers are more likely to use technology instructionally than their 
elementary colleagues. 



Method 

Data were collected from 508 teachers employed by the Mansfield Independent School District, a suburban 
PK-12 school system located within the Dallas-Fort Worth, Texas, metropolitan area. The target population was 
selected for several reasons. Although the configuration of grade levels varies in schools across the United States, 
most configurations include grades PK-4 at the elementary level as do the schools in the subject district. The study 
included PK-4 teachers, reading specialists, and teachers of special education, music, Spanish, art, and physical 
education. Teachers who were absent on the day of testing, or opted not to participate in the study, were considered 
non-participants. The results of nine surveys were eliminated from analysis because four were completed by 
paraprofessionals, two were completed by administrators, and three included incorrectly completed GSDs. 

Individual surveys that were not completed properly or did not reflect a dominant learning style were not 
included in the analysis. As a result, 499 teachers (21 males, 478 females, one of unknown gender) were included in 
the study. Among them, 253 had a bachelor’s degree only, 135 had taken some graduate classes, and 99 had earned 
a master’s degree. Six reported having completed some classes at the doctoral level. Their ages ranged from 20 to 51 
and their years of teaching experience ranged from 1 year to 20 years. 

Measures 

Learning style was assessed using the research version of the GSD, which determines how an individual 
prefers to receive and process information (Gregorc, 1982). The GSD included ten sets of four words, which 
participants were asked to rank in order of preference. Each word was identified with a learning style. Sub scores for 
each learning style were tabulated. A learning style was considered dominant if the score was greater than 27 on a 
learning styles subscale (Gregorc, 1982b). 

An adapted version of the Snapshot Survey (Norris & Solloway, 1999) was used to assess the needs of 
teachers related to educational technology, their beliefs about technology, the degree to which they have adopted it 
for use in the classroom, and the degree of their expertise with commonly used applications. The Snapshot Survey 
consists of three sections. It includes a twelve item needs assessment, an eight item assessment of the respondents’ 
beliefs about technology, the one item Concern-Based Adoption Model Level of Use [CBAM LOU] assessment 
regarding the degree to which teachers have adopted technology in the classroom, and a sixteen item assessment of 
their software expertise on commonly used applications (Norris & Solloway, 1999). All of the items, with the 
exception of the CBAM LOU, are scored using a five point Likert type scale (Norris & Solloway, 1999). 

The wording of item two on the Needs Assessment section of the Snapshot Survey was modified to render it 
more appropriate for the population surveyed. Respondents were asked about the degree to which they “need more 
time to change the curriculum to better incorporate the technology” (Norris & Solloway, 1999). The Texas Essential 
Knowledge and Skills [TEKS] are the state required learning objectives for all students in the state of Texas and 
cannot be changed. Because the wording might lead to the impression among the respondents that they were being 
asked whether they needed to change the TEKS, the item was rewritten to read, “need more time to plan lessons to 
better incorporate the technology.” Cronbach’s alpha for the twelve scaled items on the needs component of the 
Snapshot Survey was calculated as .80. 

The Assessment of Software Expertise was also modified to include more recent software packages that are 
used for instructional and administrative purposes on elementary campuses in the Mansfield Independent School 
District. This modified form of the Assessment of Software Expertise portion of the Snapshot Survey, which includes 
sixteen items, had a Cronbach’s alpha of .83. 

The Technology in Education Competency Survey [TCES] was used to assess the degree to which teachers 
perceived themselves to be proficient in the use of technology for educational purposes (Christensen, 1999). It is a 
self assessment that contains nine items and has demonstrated “an internal consistency reliability estimate of .92 
across 188 pre-service educators and 40 university faculty” (Knezek, et al, 2000, p. 39, citing Christensen & 
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Knezek, 2000). The Institute for the Integration of Technology into Teaching and Learning [IITTL] web site lists the 
source of the criteria as a survey published by the International Society for Technology in Education [ISTE] (1999). 
Knezek et al (2000) list the source of the criteria assessed on the survey as the National Council for the 
Accreditation of Teacher Education [NCATE]. Scores for each subscale on the Snapshot Survey, and for the TCES 
as a whole, were tabulated. 

Data Analysis 

All statistical data were analyzed using Statistical Program for the Social Sciences ® [SPSS] data analysis 
software for Windows, Version 10.0, 2000. In order to describe the targeted population, descriptive data were 
generated for the following demographic items: years of teaching experience, gender, years of teaching experience 
in the Mansfield ISD, teaching assignment, number of credit hours of technology related instruction in teacher 
certification program, number of clock hours of technology related instruction in other district (if applicable), 
number of clock hours of technology instruction in the Mansfield ISD, age, highest level of education, and learning 
style. Descriptive data also included frequencies, means, sample sizes, standard deviations, and variances. The .05 
alpha level was applied to all data as the standard for significance. 

A one-way analysis of variance [ANOVA] was used to test hypotheses one, two, three, four, and five. 
Levine’s Homogeneity of Variance Test was conducted to ensure that differences among the means, related to the 
measure used, were not sufficiently similar to permit comparison. 

Results 

ANOVA was used to determine whether significant differences existed between groups of teachers, grades 
PK-4, with respect to their mean scores on self-assessments related to the following variables: technology-related 
needs, beliefs about technology, levels of adoption of technology, software expertise, and technology competencies. 
These procedures indicated that there were significant differences between groups with respect to the needs, stages 
of adoption, software expertise, and technology competency variables. Tukey HSD post hoc tests were performed 
for each variable in order to determine which groups, if any, differed significantly from one another. In several 
cases, the Tukey HSD tests were not effective at identifying differences between specific groups for which the 
analyses of variance had identified differences. As a result, Fisher LSD tests were conducted on each variable for 
which the ANOVA found significant differences between the learning style groups. 

For the needs variable, the results of the Tukey HSD tests indicated that there were significant differences 
between the AS and CS groups and between the AS and AR groups. The Fisher LSD post hoc tests also found 
significant differences between the CS and AS groups and the AS and AR groups. 

The ANOVA indicated that there was no significant difference between the four learning style groups on 
the beliefs variable. No additional analysis was conducted with respect to this variable. The ANOVA detected a 
significant difference between the learning style groups on the stages of adoption variable. The Tukey HSD post hoc 
tests failed to identify significant differences between the individual groups. The set of Fisher LSD tests, however, 
identified significant differences between the following groups: AS and AR; AR and CR; AS and CR. 

The ANOVA indicated a significant difference between the learning style groups on the software expertise 
variable (p=.014). The Tukey HSD post hoc tests failed to identify significant differences between the individual 
groups. Fisher LSD post hoc tests, however, identified significant differences between the following groups: AS and 
AR; AR and CR. 

The ANOVA indicated a difference between the learning style groups on the technology competency 
variable. The Tukey HSD post hoc tests failed to identify any significant differences between the groups. The 
Fisher LSD post hoc tests, however, identified differences between the following groups: CS and AS; AS and AR; 
AR and CR. 



Discussion 

Data analysis procedures were complicated by the failure of the Tukey HSD post hoc test to pinpoint 
differences among specific groups that had been identified by analyses of variance. As a result, the Fisher LSD was 
utilized as a post hoc test to identify differences between groups on variables for which an ANOVA had identified a 
significant difference. The findings for each variable are discussed with respect to the findings of each test on each 
variable. 
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On the needs variable, both the Tukey HSD and the Fisher LSD tests indicated that there was a significant 
difference between the means scores of the AS, CS, and AR groups. Only the Fisher LSD test detected a difference 
between the AS and CR groups. The mean scores for the AS group were higher than those for the other groups. 

For the stages of adoption variable, the Tukey HSD failed to identify any differences between groups. 
Fisher LSD tests indicated that there was a significant difference between the AS and AR groups and the AR and 
CR groups. The mean scores for the AS group were higher than those of the AR group. The CR scores were 
significantly higher than the CR group. 

For the stages of adoption variable, the Tukey HSD and Fisher LSD tests indicated that there was a 
significant difference between the CS and AS groups. The Fisher LSD found differences for this variable between 
the AS and CS groups, the AS and AR groups, and AR and CR groups. The scores for the AS group were 
significantly higher than those of the AR and CS groups. The CR scores were significantly higher than those of the 
AR group. 

It was determined that AS teachers reported significantly lower needs than their CS, AR, and CR 
colleagues. AS teachers had significantly higher stages of adoption than AR teachers, and CR teachers were 
determined to have significantly higher stages of adoption than AR learners. In addition, it was determined that AS 
teachers reported significantly higher levels of technology competency than CS or AR teachers. CR teachers also 
reported significantly higher levels of competency than AR teachers. 

Summary 

Perhaps it is possible that an approach to technology staff development that takes into account the 
uniqueness of teachers might have a favorable impact on teaching and learning with technology. The AS group, 
which accounts for the smallest percentage of teachers in the sample (7.6%), had the fewest technology-related 
needs, highest stages of adoption, and more developed technology competencies. Further research is needed to 
determine why AS teachers, as noted in several previous studies, appear to be more adept and enthusiastic users of 
computers than their CS and AR counterparts. 

The findings of the current study, especially when coupled with the findings of previous research, also 
suggest that AR and CS learners may struggle more with using technology in instructional settings. Compared to AS 
teachers, CS and/or AR teachers had scores that reflected greater need or lower proficiency on each of the variables 
studied in the course of this investigation. This finding has significant implications. CS and AR groups jointly 
accounted for 67.2% of all teachers in the sample used in the current study. If the results of the current study can be 
generalized to teachers of grades PK-4, the majority of teaching personnel who are responsible for teaching students 
about technology may be less adept at performing this function than a much smaller group of their colleagues. 

Accordingly, the following recommendations have been made for further research regarding the relationship 
between learning styles, as identified by the GSD, and technology-related attributes of teachers: 

1 . Explore the reasons that AS and, to a lesser degree, CR learners appeared to have significantly fewer needs 
and significantly higher stages of adoption and technology competencies than CS and/or AR learners. 

2. Conduct a study that places participants in treatment groups based on their dominant learning styles. 
Investigate whether instructional strategies designed for their dominant learning styles have an impact on 
their mastery of technology-related knowledge and skills. 

3. Replicate this study with a group of secondary teachers to determine if there are any differences between 
teachers in intermediate, middle, and high school settings. 

It is hoped that by continuing to investigate the learning styles of teachers, technology staff development can be 
improved. If staff development strategies can be tailored to meet the needs of the teachers who attend them, perhaps 
teachers will become more effective in facilitating instruction using instructional technology. 
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An Assessment of Undergraduate Student’s Mobility Skills and Needs in 

Curriculum Delivery 

Dale Higginbotham 
University of Houston 

Abstract 

The most important characteristic of our society in the last three decades may well be the incredible speed 
with which it changes. Scientific, technological, social and cultural innovation are taking place at such a fast pace, 
students constantly need to revise their skills in order to adapt to the changing technologies. In our day-to-day lives 
communication mobility not only effects interpersonal communication and entertainment but has become equally 
important in succeeding in ones pursuit of higher education. 

Today we are living in a wireless world. Mobile communication devices (e.g., iPods, cell phones, laptop 
computers, PDAs) are expanding the scope of education and learning. As Bryan Alexander noted in his article. 

Going nomadic: Mobile learning in higher education, such devices foster mobile learners, or "m-leamers”— students 
who are constantly connected. M-leamers engage in ”m-leaming,'’ rapid, just-in-time learning, fueled by feedback 
and the capability for shared and multi-way communications. Thus, students can instantly construct their own 
learning environments (Alexander 2004). 

But along with the positive aspects of technology also comes the negative. How much can one student 
absorb while being involved in other activities? Linda Stone, a former Microsoft researcher, characterizes ours as an 
era of “continuous partial attention.” At the extreme end are the traditional students instant-messaging while they 
talk on the cellphone, download music and do homework. But non-traditional students too live with all systems go, 
interrupted and distracted, scanning everything, multi-technological-tasking everywhere. We suffer from the 
illusion, says Stone that we can expand our “personal bandwidth”, connecting to ever more important information. 
Instead, we end up over stimulated, overwhelmed and unfulfilled. Continuous partial attention inevitably feels like a 
lack of full attention” (Stone, 2005). 

How much information can be too much information? The term “Information overload” was coined in 
1970 by Alvin Toffler in his book. Future Shock. Information overload refers to the state of having too much 
information to make a decision or remain informed about a topic. This term is normally used in conjunction with 
various forms of computer-mediated communication. Large amounts of currently available information, a high rate 
of new information being added, contradictions in available information, a low signal-to-noise ratio, and inefficient 
methods for comparing and processing different kinds of information can all contribute to this effect (Toffler, 1970). 
In a world where information overload is common, attention is a very scarce resource and there is an increasing need 
to manage it efficiently. In a connected world it is becoming very difficult to filter out the information that really 
needs our attention from that which is irrelevant to us. 

The Study 

The research data is from Phase I of an ongoing research study, funded by the University of Houston’s 
Educational Technology and University Outreach. Which involved students enrolled in multiple classes in the field 
of communication. The students attended either traditional face-to-face classes or fully online courses. Instructors 
used Podcasting and Vodcasting to deliver course lecture materials or supplement materials as an alternative channel 
to facilitate student learning. Survey research methods were used to conduct the study. The study specifically 
investigated the areas listed below related to Podcasting and Vodcasting of course content materials while 
multitasking: 

1 . The use of portable media devices (PMD) - iPods/Mp3 players to access instructional materials while 
involved in other activities 

a) The time duration for their use while multitasking 
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b) The frequency of use while multitasking 

c) The purpose they use portable devices while multitasking 

2 . Podcasting course materials 

a) The hours students spend viewing/listening to Podcasting materials while multitasking 

b) The delivery quality of Podcasting materials while multitasking 

c) The technical issues related to the Podcasting delivery while multitasking 

d) Students’ preferred Podcasting format, audio/video/still image, etc., while multitasking 

3 . Podcasting learning styles and study habits 

a) The availability of Podcasting course materials changes study habits because of their ability of use 
while multitasking 

b) Podcasting delivery format and the enhancement of learning experiences while multitasking 

c) Podcasting delivery format and personal learning style as developed during multitasking 

d) Podcasting and mobile learning perceptions while multitasking 

e) Students mobility needs 

4 . Podcasting and learning effectiveness 

a) How well does the student retain information while multitasking 

b) The helpfulness of the addition of Podcasting course material as available resources to the 
understanding of course content 

c) Podcasting and improvement of leaning outcome 

Study Results 
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Demographics 

The University of Houston is a large urban university with over 35,000 students. The majority of our 
students are non-traditional commuter students. In this study our population is made up of 62% female/38% 
male and the majority was between 20 -25 years of age (Tables 1-2). We found that a 41% majority of the 
students commuted 20 or more miles to campus (Table 6), with a typical commute between 21-50 minutes one- 
way (Table 7). 

The formats of the courses studied were: Traditional face-to-face, no online usage; traditional face-to-face; 
some online usage; hybrid/bended class - 50% or more online; and fully online with the majority of the students 
were in traditional face-to-face with some online usage (Table 8). The majority of the students accessed their 
course materials from home using high speed (Cable or DSL) internet connections (Tables 3-4). In measuring 
their computer competency a majority of 63% of the students have been using computers for eight or more 
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years (Table 5). Given the age, level of technology expertise and the technology they have at their disposal we 
concluded that our population would be extremely adaptable to new technologies. 

Podcasting being a very new technology, we asked the students if before entering their courses they new 
what a podcast was. 58% percent of the students did but only 31% had ever viewed or listened to one (Tables 
9-10). 



Use of Portable Media Devices (PMD) 

A 38% majority of the students studied responded that they use their PMDs 5 hours or less a week (Tables 
1 1& 14). When asked when the study subjects preferred to use their PMDs the majority answered that they 
used them while engaged in other activities such as exercising or driving (Tables 12 & 15). Also the majority 
of the students studied indicated that the activity that they engaged in the most with their PMDs was listening to 
music (Table 13 & 16). 

Use of Podcasting Course Materials 
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34% of the study population replied that they accessed instructional materials using their PMDs while 
involved in other activities (Table 17). But when asked if they would access instructional materials in the future 
with their PMDs a 67% majority said they would (Table 18). When asked how many hours a week they spent 
using their PMD to access instruction materials while involved in other activities an 81% majority said that they 
used it less than 5 hours a week (Table 19). This data would indicate that because this technology is so new the 
students have been conditioned to use it solely for entertainment and would need to be shown that it could have 
educational purposes especially while multitasking. 

When surveyed concerning the quality of PMDs to access instruction materials while involved in other 
activities 69% approved (Table 20). At the same time 88% responded that they did not have any technical 
issues related to their PMDs use (Table 21). When asked if the student was an auditory, visual or tactile learner 
the students responded that they were mainly visual or tactile learners (Table 31). There was an even split in 
preferences between audio, audio with image and video podcasts (Table 22). 



Podcasting Learning Styles and Study Habits 



An even split occurred when respondents were asked if they preferred podcasts to attending lectures (table 
35 & 36). Instead they preferred the combination of PMD course delivery with face-to-face classroom 
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experience (Table 37). Also the students were evenly divided when asked why they used the podcast material 
and if the podcasts changed their study habits (Table 26 & 27). Along the same lines, they were divided when 
asked that the podcasts changed their behaviors outside of the classroom when studying for exams (Table 28). 
The students reading time was not effected by including the podcasts in their course delivery (Table 38 & 39) 
and their total study time was also unaffected (Table 40 & 41). 

The majority of the students studied felt that the podcast delivery format enhanced their learning experience 
and made learning more enjoyable (Table 29 & 30). Concerning the student’s mobility needs the majority of 
the respondents felt that they preferred the mobility that PMDs provided and they found that the ability to 
review materials whenever they wanted was the most important feature (Tables 33 & 34). 




Podcasting and Learning Effectiveness 

Concerning information retention while using PMDs to access instructional materials while multitasking, 
the majority of those surveyed responded that it had a positive effect (Table 23), as well as when asked of their 
understanding of course content (Table 24). Overall the students felt the addition of the podcast course material 
would make an improvement to their learning outcome (Table 25). 

Conclusions 

In conducting our research at a large urban university where the typical student is non-traditional with 
many time restraints, we have concluded that using PMDs to access instructional materials would be most 
beneficial, especially in conjunction with their other daily activities. 

With long commutes valuable time is lost from our student’s day. This time could be better spent studying 
their course work. Also during other activities such as exercising, students could be multitasking with their 
course work as well. 

Our study has shown us that podcasting technology is easy to access and easy to use. The flexibility and 
portability of having course materials recorded allows the student to be able to review these materials wherever 
and whenever they want to and at their own pace. In the traditional lecture this information is heard once and 
then lost. With this technology the student will be able to review this information as many times as they like in 
order to reinforce it. We have found that by incorporating PMDs into a student’s course materials the student’s 
study habits will change and it will also enhance their learning experience. The format of podcasting material, 
audio/video/still image would depend on the student’s individual learning style. 

With the introduction of new technologies earlier in the student’s education our student body is more 
technologically savvy than ever before. With these skills developed at an early age our students today are more 
adaptable to new technologies and are able to converge them into their everyday lives. Because of the ability to 
multitask, podcasting engages students and increases their potential to perform better in their overall learning. 
Because of this adaptability our research indicates that the student has a greater ability to retain information 
while multitasking. 
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Students today have very specific mobility needs. Balancing their work and families while trying to obtain 
an education leaves very little leisure time. Having more tools made available to these students will create a 
better perception of their educational outcome. The better the student perceives their educational outcome the 
greater are their chances of succeed. 



Further Study 

The next phase of our study will involve students that will represent other areas of instruction taught at the 
University of Houston. Our initial study only involved students studying Communication but our next phase 
will include students majoring in Science and Mathematics as well as the Arts. We purpose to study how these 
different backgrounds will effect how the student is able to adapt and utilize this technology in their education 
Also we plan to study how the podcasting technology must be adapted inorder to fulfill the different 
instructional needs of these different areas of study. 
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Table 1 

SI :Q1 - Age Groups 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Under 20 


17 


10.6 


10.6 


10.6 


20-25 


120 


74.5 


74.5 


85.1 


26-30 


18 


11.2 


11.2 


96.3 


31-35 


2 


1.2 


1.2 


97.5 


36-40 


3 


1.9 


1.9 


99.4 


41 - above 


1 


.6 


.6 


100.0 


Total 


161 


100.0 


100.0 





Table 2 

SI - Q2: Gender 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Female 


99 


61.5 


61.5 


61.5 


Male 


62 


38.5 


38.5 


100.0 


Total 


161 


100.0 


100.0 





Table 3 

SI - Q3; What kind of internet connection(s) do you have? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Phone line dial-up 


19 


11.5 


11.5 


11.5 


High speed (Cable, DSL) 


133 


80.6 


80.6 


92.1 


Tl/LAN/WAN 


11 


6.7 


6.7 


98.8 


Other 


2 


1.2 


1.2 


100.0 


Total 


165 


100.0 


100.0 





Table 4 

S 1 - Q4; From what location do you access the internet most? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Campus 


28 


17.4 


17.5 


17.5 


Home 


124 


77.0 


77.5 


95.0 


Workplace 


8 


5.0 


5.0 


100.0 


Total 


160 


99.4 


100.0 




Missing System 


1 


.6 






Total 


161 


100.0 
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Table 5 

SI - Q5 : How long have you been using computers? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid -2 years 


1 


.6 


.6 


.6 


2-4 years 


6 


3.7 


3.8 


4.4 


5-8 years 


52 


32.3 


32.5 


36.9 


8+ years 


101 


62.7 


63.1 


100.0 


Total 


160 


99.4 


100.0 




Missing System 


1 


.6 






Total 


161 


100.0 







Table 6 

SI - Q6: How far do you live from campus? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid I live on campus 


13 


8.1 


8.1 


8.1 


1-5 miles 


22 


13.7 


13.7 


21.7 


6-10 miles 


20 


12.4 


12.4 


34.2 


11-15 miles 


11 


6.8 


6.8 


41.0 


16-20 miles 


29 


18.0 


18.0 


59.0 


20+ miles 


66 


41.0 


41.0 


100.0 


Total 


161 


100.0 


100.0 





Table 7 

SI - Q7 : How long is your typical commute to campus? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid -10 minutes 


33 


20.5 


20.8 


20.8 


11-20 minutes 


24 


14.9 


15.1 


35.8 


21-30 minutes 


25 


15.5 


15.7 


51.6 


31-40 minutes 


40 


24.8 


25.2 


76.7 


41-50 minutes 


25 


15.5 


15.7 


92.5 


60+ minutes 


12 


7.5 


7.5 


100.0 


Total 


159 


98.8 


100.0 




Missing System 


2 


1.2 






Total 


161 


100.0 







104 

















Table 8 

SI - Q8: What is the format of the course you are currently taking? 











Valid 








Frequency 


Percent 


Percent 


Cumulative Percent 


Valid 


Traditional face-to-face class, no 
online usage 

Traditional face-to-face class, some 
online usage 


22 

101 


13.7 

62.7 


13.8 

63.1 


13.8 

76.9 




Hybrid/Blended class, (50% or more 
online) 


9 


5.6 


5.6 


82.5 




Fully online course 


28 


17.4 


17.5 


100.0 




Total 


160 


99.4 


100.0 




Missing 


System 


1 


.6 






Total 




161 


100.0 







Table 9 

S2 - Q1 : Did you know what a podcast was before you took this course? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Yes 


73 


57.5 


57.5 


57.5 


No 


54 


42.5 


42.5 


100.0 


Total 


127 


100.0 


100.0 





Table 10 

S2 - Q2: Did you ever listen to a podcast before you took this course? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Yes 


40 


31.5 


31.5 


31.5 


No 


87 


68.5 


68.5 


100.0 


Total 


127 


100.0 


100.0 





Table 11 

S2 - Q4: How many hours do you use your iPod a week? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Over 50 hours a week 


1 


1.1 


1.1 


1.1 


40-50 hours a week 


4 


4.2 


4.2 


5.3 


30-40 hours a week 


4 


4.2 


4.2 


9.5 


20-30 hours a week 


11 


11.6 


11.6 


21.1 


10-20 hours a week 


22 


23.2 


23.2 


44.2 


5-10 hours a week 


18 


18.9 


18.9 


63.2 


Less than 5 hours 


35 


36.8 


36.8 


100.0 


Total 


95 


100.0 


100.0 
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Table 12 

S2 - Q5: When do you prefer to use your iPod? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid At home 


42 


18.4 


18.4 


18.4 


Driving in the car 


46 


20.2 


20.2 


38.6 


Walking or jogging 


62 


27.2 


27.2 


65.8 


In public location 


37 


16.2 


16.2 


82.0 


Traveling 


41 


18.0 


18.0 


100.0 


Total 


228 


100.0 


100.0 





Table 13 



S2 - Q6: What activities do you do most with your iPod? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Listen to music 


85 


65.9 


65.9 


65.9 


Watch video 


14 


10.9 


10.9 


76.7 


Listen to radio 


10 


7.8 


7.8 


84.5 


Access RSS news 


1 


.8 


.8 


85.3 


Access course materials 


10 


7.8 


7.8 


93.0 


Access Podcast news 


9 


7.0 


7.0 


100.0 


Total 


129 


100.0 


100.0 





Table 14 

S3 - Q2: How many hours do you use your MP3 player a week? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Over 50 hours a week 


2 


3.6 


3.6 


3.6 


40-50 hours a week 


1 


1.8 


1.8 


5.5 


20-30 hours a week 


11 


20.0 


20.0 


25.5 


1 0-20 hours a week 


9 


16.4 


16.4 


41.8 


5-10 hours a week 


11 


20.0 


20.0 


61.8 


Less than 5 hours 


21 


38.2 


38.2 


100.0 


Total 


55 


100.0 


100.0 





Table 15 

S3 - Q3: When do you prefer to use your MP3 player? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid At home 


22 


20.2 


20.2 


20.2 


Driving in the car 


18 


16.5 


16.5 


36.7 


Walking or jogging 


29 


26.6 


26.6 


63.3 


In public location 


22 


20.2 


20.2 


83.5 


Traveling 


18 


16.5 


16.5 


100.0 


Total 


109 


100.0 


100.0 
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Table 16 

S3 - Q4: What activities do you do most with your MP3 player? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Listen to music 


50 


66.7 


66.7 


66.7 


Watch video 


7 


9.3 


9.3 


76.0 


Listen to radio 


10 


13.3 


13.3 


89.3 


Access RSS news 


1 


1.3 


1.3 


90.7 


Access course materials 


4 


5.3 


5.3 


96.0 


Access Podcast news 


3 


4.0 


4.0 


100.0 


Total 


75 


100.0 


100.0 





Table 17 

S4 - Q 1 : Did you use your iPod/MP3 player to access instructional materials while involved in other activities? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Yes 


47 


33.8 


33.8 


33.8 


No 


92 


66.2 


66.2 


100.0 


Total 


139 


100.0 


100.0 





Table 18 

S4 - Q2: If not, would you use iPod/MP3 player to access instructional materials while involved in other activities if 

it were available to you? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Yes 


75 


67.6 


67.6 


67.6 


No 


36 


32.4 


32.4 


100.0 


Total 


111 


100.0 


100.0 





Table 19 

S4 - Q3: How many hours did you spend in using your iPod/MP3 player to access instructional materials while 

involved in other activities? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid 40-50 hours a week 


1 


.8 


.8 


.8 


30-40 hours a week 


1 


.8 


.8 


1.6 


20-30 hours a week 


2 


1.6 


1.6 


3.3 


10-20 hours a week 


3 


2.4 


2.4 


5.7 


5-10 hours a week 


16 


13.0 


13.0 


18.7 


Less than 5 hours 


100 


81.3 


81.3 


100.0 


Total 


123 


100.0 


100.0 
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Table 20 

S4 - Q6: How would you rate the quality of portable media devices (e.g. iPod, MP3 player, etc.) to access 
instructional materials while involved in other activities? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Excellent 


29 


22.3 


22.3 


22.3 


Good 


61 


46.9 


46.9 


69.2 


Poor 


4 


3.1 


3.1 


72.3 


Bad 


2 


1.5 


1.5 


73.8 


N/A 


34 


26.2 


26.2 


100.0 


Total 


130 


100.0 


100.0 





Table 21 

S4 - Q8: Did you have any technical issues related to the use of portable media devices to access instructional 

materials while involved in other activities? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Yes 


14 


12.5 


12.5 


12.5 


No 


98 


87.5 


87.5 


100.0 


Total 


112 


100.0 


100.0 





Table 22 

S4 - Q9: What is your preferred format of podcasts when you use portable media devices to access instructional 

materials while involved in other activities? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Audio 


41 


33.9 


33.9 


33.9 


Audio with image 


37 


30.6 


30.6 


64.5 


Video 


43 


35.5 


35.5 


100.0 


Total 


121 


100.0 


100.0 





Table 23 

S4 - QIO: How well did you retain information while using portable media devices to access instructional materials 

while involved in other activities? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Very Well 


9 


6.6 


6.9 


6.9 


Well 


54 


39.4 


41.5 


48.5 


N/A 


49 


35.8 


37.7 


86.2 


Fittle 


16 


11.7 


12.3 


98.5 


Not at all 


2 


1.5 


1.5 


100.0 


Total 


130 


94.9 


100.0 




Missing System 


7 


5.1 






Total 


137 


100.0 
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Table 24 

S4 - Q1 1 : How helpful did you feel the addition of podcasting course material was to your understanding of course 

content? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Very helpful 


21 


15.3 


15.4 


15.4 


Helpful 


63 


46.0 


46.3 


61.8 


N/A 


35 


25.5 


25.7 


87.5 


Little helpful 


12 


8.8 


8.8 


96.3 


No help at all 


5 


3.6 


3.7 


100.0 


Total 


136 


99.3 


100.0 




Missing System 


1 


.7 






Total 


137 


100.0 







Table 25 

S4 - Q12: How helpful did you feel the addition of podcasting course material was to the improvement of your 

learning outcome? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Very helpful 


15 


10.9 


11.3 


11.3 


Helpful 


60 


43.8 


45.1 


56.4 


N/A 


36 


26.3 


27.1 


83.5 


Little helpful 


13 


9.5 


9.8 


93.2 


No help at all 


9 


6.6 


6.8 


100.0 


Total 


133 


97.1 


100.0 




Missing System 


4 


2.9 






Total 


137 


100.0 







Table 26 

S5 - Q6: 1 viewed podcast course materials due to following reason(s) 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Required content in other formats 


16 


3.8 


3.8 


3.8 


Missed lectures 


59 


14.0 


14.0 


17.8 


Making up missing notes from class 


46 


10.9 


10.9 


28.7 


Course content preview 


21 


5.0 


5.0 


33.7 


Course content review 


58 


13.8 


13.8 


47.5 


Convenient access 


62 


14.7 


14.7 


62.2 


Flexibility 


61 


14.5 


14.5 


76.7 


Portability 


42 


10.0 


10.0 


86.7 


Ease of use 


56 


13.3 


13.3 


100.0 


Total 


421 


100.0 


100.0 
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Table 27 

S6 - Q1 : The availability of course material podcasts changed my study habits. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


7 


4.6 


4.6 


4.6 


Agree 


42 


27.6 


27.8 


32.5 


N/A 


45 


29.6 


29.8 


62.3 


Disagree 


51 


33.6 


33.8 


96.0 


Strongly Disagree 


6 


3.9 


4.0 


100.0 


Total 


151 


99.3 


100.0 




Missing System 


1 


.7 






Total 


152 


100.0 







Table 28 

S6 - Q2; Podcasting changed my behaviors outside of the classroom when studying for exams. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


5 


3.3 


3.3 


3.3 


Agree 


41 


27.0 


27.0 


30.3 


N/A 


43 


28.3 


28.3 


58.6 


Disagree 


57 


37.5 


37.5 


96.1 


Strongly Disagree 


6 


3.9 


3.9 


100.0 


Total 


152 


100.0 


100.0 





Table 29 

S6 - Q3: The podcast delivery format enhanced my learning experiences. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


11 


7.2 


7.3 


7.3 


Agree 


67 


44.1 


44.7 


52.0 


N/A 


44 


28.9 


29.3 


81.3 


Disagree 


24 


15.8 


16.0 


97.3 


Strongly Disagree 


4 


2.6 


2.7 


100.0 


Total 


150 


98.7 


100.0 




Missing System 


2 


1.3 






Total 


152 


100.0 
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Table 30 

S6 - Q4: Podcast delivery format made learning more enjoyable. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


13 


8.6 


8.7 


8.7 


Agree 


65 


42.8 


43.3 


52.0 


N/A 


46 


30.3 


30.7 


82.7 


Disagree 


22 


14.5 


14.7 


97.3 


Strongly Disagree 


4 


2.6 


2.7 


100.0 


Total 


150 


98.7 


100.0 




Missing System 


2 


1.3 






Total 


152 


100.0 







Table 31 

S6 - Q5: Which of the following statements best describes your learning style? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Auditory learner 


23 


15.1 


15.2 


15.2 


Visual learner 


66 


43.4 


43.7 


58.9 


Tactile learner 


62 


40.8 


41.1 


100.0 


Total 


151 


99.3 


100.0 




Missing System 


1 


.7 






Total 


152 


100.0 







Table 32 



S6 - Q6: Which of the following podcast formats suited your personal learning style? 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Audio podcasts 


27 


19.7 


20.1 


20.1 


Podcasts with audio and still images 


29 


21.2 


21.6 


41.8 


Video podcasts 


78 


56.9 


58.2 


100.0 


Total 


134 


97.8 


100.0 




Missing System 


3 


2.2 






Total 


137 


100.0 
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Table 33 

S6 - Q7: Which of the following features of podcasts did you find most valuable? 











Valid 


Cumulative 






Frequency 


Percent 


Percent 


Percent 


Valid 


Ability to review materials wherever I 
wanted. 


29 


21.2 


22.1 


22.1 




Ability to review materials whenever I 
wanted. 


80 


58.4 


61.1 


83.2 




Ability to review materials at my own pace. 


22 


16.1 


16.8 


100.0 




Total 


131 


95.6 


100.0 




Missing 


System 


6 


4.4 






Total 




137 


100.0 







Table 34 

S6 - Q8: 1 prefer the mobility that the iPod/MP3 player provides. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


41 


27.3 


27.7 


27.7 


Agree 


67 


44.7 


45.3 


73.0 


N/A 


33 


22.0 


22.3 


95.3 


Disagree 


6 


4.0 


4.1 


99.3 


Strongly Disagree 


1 


.7 


.7 


100.0 


Total 


148 


98.7 


100.0 




Missing System 


2 


1.3 






Total 


150 


100.0 







Table 35 

S6 - QIO: I prefer podcasts for course delivery to attending lectures. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


22 


14.4 


14.4 


14.4 


Agree 


40 


26.1 


26.1 


40.5 


N/A 


30 


19.6 


19.6 


60.1 


Disagree 


37 


24.2 


24.2 


84.3 


Strongly Disagree 


24 


15.7 


15.7 


100.0 


Total 


153 


100.0 


100.0 
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Table 36 

S6 - Q1 1 : 1 did not use the iPod/MP3 player at all because I learn better from the face-to-face classroom experience 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


16 


10.7 


10.7 


10.7 


Agree 


31 


20.8 


20.8 


31.5 


N/A 


47 


31.5 


31.5 


63.1 


Disagree 


51 


34.2 


34.2 


97.3 


Strongly Disagree 


4 


2.7 


2.7 


100.0 


Total 


149 


100.0 


100.0 





Table 37 

S6 - Q12: 1 prefer the combination of iPod/MP3 player course delivery with the face-to-face classroom experience. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


28 


18.8 


18.9 


18.9 


Agree 


69 


46.3 


46.6 


65.5 


N/A 


20 


13.4 


13.5 


79.1 


Disagree 


24 


16.1 


16.2 


95.3 


Strongly Disagree 


7 


4.7 


4.7 


100.0 


Total 


148 


99.3 


100.0 




Missing System 


1 


.7 






Total 


149 


100.0 







Table 38 

S6 - Q14: My reading time decreased with the addition of podcasts. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


6 


4.0 


4.1 


4.1 


Agree 


39 


26.0 


26.5 


30.6 


N/A 


49 


32.7 


33.3 


63.9 


Disagree 


44 


29.3 


29.9 


93.9 


Strongly Disagree 


9 


6.0 


6.1 


100.0 


Total 


147 


98.0 


100.0 




Missing System 


3 


2.0 






Total 


150 


100.0 
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Table 39 

S6 - Q15: My reading time increased with the addition of podcasts. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


1 


.7 


.7 


.7 


Agree 


17 


11.3 


11.5 


12.2 


N/A 


54 


36.0 


36.5 


48.6 


Disagree 


58 


38.7 


39.2 


87.8 


Strongly Disagree 


18 


12.0 


12.2 


100.0 


Total 


148 


98.7 


100.0 




Missing System 


2 


1.3 






Total 


150 


100.0 







Table 40 

S6 - Q16: My studying time decreased with the addition of podcasts. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


4 


2.7 


2.7 


2.7 


Agree 


31 


20.7 


20.8 


23.5 


N/A 


49 


32.7 


32.9 


56.4 


Disagree 


60 


40.0 


40.3 


96.6 


Strongly Disagree 


5 


3.3 


3.4 


100.0 


Total 


149 


99.3 


100.0 




Missing System 


1 


.7 






Total 


150 


100.0 







Table 41 

S6 - Q17: My studying time increased with the addition of podcasts. 





Frequency 


Percent 


Valid Percent 


Cumulative Percent 


Valid Strongly Agree 


2 


1.3 


1.3 


1.3 


Agree 


27 


18.0 


18.0 


19.3 


N/A 


52 


34.7 


34.7 


54.0 


Disagree 


56 


37.3 


37.3 


91.3 


Strongly Disagree 


13 


8.7 


8.7 


100.0 


Total 


150 


100.0 


100.0 
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Abstract 

The purpose of this research was to manipulate the component of confidence found in 
Keller’s ARCS Model to enhance the confidence and performance of undergraduate students 
enrolled in an online course at a Texas University. This study also tested whether the confidence 
tactics had any unintentional effect on the remaining attention, relevance, and satisfaction 
subscales of the ARCS model and learners’ overall motivation for the class and the instructional 
materials. The results indicated that the treatment group showed statistically greater gains than 
the control group in terms of learner confidence on the CIS but not the IMMS. The treatment 
group outperformed the control group on all of the individual posttest measures and on the 
overall mean performance score. The results showed no statistically significant difference on the 
attention subsection. However, statistically significant differences were noted for the relevance 
and satisfaction subscales. There was also a statistically significant difference in overall learner 
motivation as measured on both surveys. 

Introduction 

Motivation is a highly important yet under-researched aspect of learning. Means, 
Jonassen, and Dwyer (1997) cited studies showing that motivation accounted for 16% to 38% of 
the variations in overall student achievement. However, an extensive review of the literature 
leads one to concur that there is a noted lack of research concerning the motivational needs of 
learners (Astleitner & Keller, 1995; Gabrielle, 2003; Means, Jonassen & Dwyer, 1997; Shellnut, 
Knowlton & Savage, 1999; Visser & Keller, 1990). 

This is particularly true in terms of computer and web-based instruction. Keller (1999a) 
noted that self-directed learning environments, like distance education classes, posed greater 
challenges to learner motivation than their face-to-face counterparts. Song and Keller (2001) 
advised that continued problems with learner motivation in web or site-based Computer Assisted 
Instruction (CAI) were often the result of incorrect assumptions on the part of instructional 
designers that motivation, if taken into account at all, was assumed to be already present in the 
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CAL They also noted that with the widespread use of computers in education, one could no 
longer depend on the “novelty effect” of technology to stimulate learner motivation. 

Distance education environments provide unique challenges for instructors and designers 
who wish to motivate students. Traditional distance learning models stress the independence of 
the learner (Downs & Moller, 1999; Moore, 1989) and the privatization of the learning 
environment (Keegan, 1986; Moller, et. al. 2005). Such student-centered, independent learning 
requires a strong sense of motivation and confidence. Researchers may not being paying enough 
attention to the motivational requirements of online learners. 

For example, in an in-depth study of the proceedings of the World Conferences of the 
International Council for Distance Education from 1988 to 1995, Visser, Plomp, Amirault, and 
Kuiper (2002) found that only six of 801 studies addressed motivational concerns of online 
learners. They also noted that a disturbing “trend in the lack of attention paid to motivation in 
distance education is present in some of the recently published specialized handbooks in the field 
of distance education” (p. 95). 

Keller ’s ARCS Model and Previous Studies 

To stimulate and manage student motivation to learn, Keller (1987a, 1987b, 1987c) 
created the ARCS Model of Motivation. ARCS stands for Attention, Relevance, Confidence and 
Satisfaction and serves as the framework for the motivational and confidence-enhancing tactics 
found in this study. 

Keller’s ARCS model enjoys wide support in the literature, and a number of researchers 
attest to its reliability and validity in many different learning and design environments. For 
example, ARCS research can be found concerning the traditional classroom (Bickford, 1989; 
Klein & Freitag, 1992; Means, Jonassen & Dwyer, 1997; Moller, 1993; Naime-Diefenbach, 

1991; Small & Gluck, 1994; Visser & Keller, 1990), Computer Assisted Instruction (Asteitner & 
Keller, 1995; Bohlin & Milheim, 1994; ChanLin, 1994; Lee & Boling, 1996; Shelnut, Knowlton 
& Savage, 1999; Song, 1998; Song & Keller, 1999; Suzuki & Keller, 1996), blended learning 
environments (Gabrielle, 2003), and online, distant, and web-based classrooms (Chyung, 2001; 
Huett, 2006; Song, 2000; Visser, 1998). In fact. Means, Jonassen, and Dwyer (1997) called 
Keller’s ARCS model the “only coherent and comprehensive instructional design model 
accommodating motivation” (p. 5). 

The ARCS model was initially predicated on the Expectancy Value Theory based on the 
work of Tolman (1932) and Lewin (1938). The Expectancy Value Theory essentially states that 
learners pursue activities they value and in which they expect to succeed (Keller, 1987c). The 
ARCS model is an attempt to synthesize behavioral, cognitive, and affective learning theories 
and demonstrate that learner motivation can be influenced through external conditions such as 
instructional materials (Moller, 1993). 

To quote Keller (1987a), “motivational interventions can be focused within a general 
category, or specific subcategory of the model” (p. 6). However, there is insufficient evidence to 
support such claims that learner motivation can be isolated or compartmentalized into separate 
categories. Studies of ARCS-enhanced instructional materials have returned inconsistent results 
on the individual subsections as well as on the overall measure of learner motivation. Means, 
Jonassen, and Dwyer (1997) found that “there is inconsistent evidence that each of the factors 
operates independently, that learners’ motivations can be decomposed and isolated, or that 
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changes in one motivational state have an inconsequential effeet on others” (p.6). For example, 
Naime-Diefenbach (1991) specifieally targeted increases in learner attention and eonfidenee and 
claims her study “validated the attention eomponent of the ARCS model under eontrolled 
conditions” (p.50). However, she was not successful in isolating and validating confidence. 

Also, Moller (1993) was unsueeessful in isolating and validating increases in eonfidenee 
for the treatment group in a study speeifically designed to do so. Though overall motivation, 
performanee, and self-direeted learning were the targets of Gabrielle’s (2003) study, she found 
mixed returns on the eonfidenee subseetion. Her results indicated a significant difference 
between groups as regards eonfidenee on one survey measure but not the other. In a study 
designed only to enhance the relevanee subseetion of the ARCS model, Babe (1995) also found 
statistieally significant differences between groups for all subsections (attention, relevance, 
eonfidenee, and satisfaetion). 

The results of this study (and those mentioned above) eontinue to ehallenge researehers’ 
assumptions (Keller, 1987a; Naime-Diefenbach, 1991) that individual components of the ARCS 
model ean be isolated for improvement. 

The major foeus of this study was to determine whether eonfidenee eould be speeifieally 
targeted for improvement and whether improvements in eonfidenee would translate into overall 
motivation and performance gains. The underlying assumptions are that confidence is a highly 
important aspeet of motivation, that it ean be manipulated through external factors, and that it 
has an effect on learner performanee. 

Confidenee 

Confidence has been deseribed in the literature as a personality trait (McKinney, 1960). 
However, confidence is more universally aceepted as situation-specifie, and it can therefore be 
manipulated by internal and external faetors (Keller, 1979; Moller 1993). 

In his development of Social Learning Theory, Rotter (1954) argued that people have a 
tendency to ascribe their failures or sueeesses to internal or external faetors: he found that people 
tend to pursue that which brings about the most rewarding consequence, whieh he called 
expectancy. Bandura (1977; 1986) elaborated on this eoncept when he explained that 
individuals’ expeetancy is related to their estimate of the outeome of a given behavior. He used 
the term self-efficacy to deseribe the belief that one’s abilities and knowledge are sufficient to be 
suceessful at a given task: learners who expeet to sueceed demonstrate more confidence than 
learners who expect to fail. However, a learner may still possess eonfidenee without the 
guarantee of sueeess, as long as the ehallenge is “within acceptable boundaries” (Naime- 
Diefenbach, 1991, p. 12). 

Building on the work of Rotter, Bandura, and others, Keller defines eonfidenee as 
“helping the learners believe/feel that they will sueeeed and eontrol their success” (Keller, 

1987a, p. 2). Confidenee is the interplay between desire for sueeess and fear of failure. These 
opposing forces vie for control over the learning experienee. To better understand the role of 
confidence, Keller and Suzuki (1988) eharaeterize its three most important dimensions: 
pereeived competence, pereeived eontrol, and expectaney for sueeess. 

Perceived Competence 

Confidence is about self-pereeption: of one’s abilities and control within the learning 
context. Learners who believe in their potential success are more likely to exert the effort 
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required to be sueeessful. Despite the fact that learner expectations are not always realistically 
aligned with learner abilities, expectations can still positively influence outcomes (Bickford, 
1989). 

Students with a poor perception of their abilities may become anxious and perform less 
well than their counterparts with higher confidence in their abilities (Naime-Diefenbach, 1991). 
Moller (1993) describes learners with high anxiety as often “misdirecting effort from learning to 
task-irrelevant concerns. Learners high in anxiety are often low in self-esteem and, as such, 
avoid evaluative situations” (p. 7). In contrast, learners with normal anxiety levels feel more 
confident and motivated in situations where they must be evaluated. 

Perceived Control 

When learners believe their efforts and decisions have real consequences, they feel more 
confident (Bandura, 1977; Keller & Suzuki, 1988). This fosters a higher internal locus of control 
and a greater sense of self-pride and accomplishment (Moller, 1993). In contrast, learners who 
believe luck or other uncontrollable outside forces are in charge of their successes or failures 
tend to feel more helpless and unconfident, and perform at lower levels. Keller (1979) finds that 
locus of control is more closely related to “attitudes toward performance than to actual 
performance” (p. 31). 

According to Keller and Suzuki (1988), “features in the instruction that promote feelings 
of personal control over outcomes will help develop confidence and persistence” (p. 405). This is 
supported by researchers such as Carroll (1963), Bloom (1976), and Kinzie and Sullivan (1989) 
who recommend allowing learners to control the pace of instruction. However, research is 
mixed about how much control is actually beneficial to learners (Klein & Keller, 1990). 

Steinberg (1989) cited numerous studies that showed that learners with little prior knowledge of 
the subject matter were likely to perform poorly with increased learner control. 

Keller (1987a) suggests one strategy for fostering control is to give students knowledge 
of what is expected of them. However, this is not enough to guarantee confidence: while learners 
may understand what steps are necessary to complete an assigned task, without the confidence in 
their ability to successfully complete those tasks, they may not perform as well as they should 
(Moller, 1993). The concept of control may be particularly relevant to distance learning 
environments. Roblyer (1999) found that students who chose distance education classes over 
face-to-face classes often did so out of a greater desire or need for control over their own 
learning outcomes. 

Expectancy for Success 

Learners’ expectations or beliefs can influence outcomes. For example, if the learner 
believes he will be successful at a given task, such belief may result in greater effort expended 
and improve success. Learners with such expectancy for success also possess higher motivation 
than learners who expect failure (Naime-Diefenbach, 1991). 

The learner who expects failure may also evince learned helplessness (Keller, 1979; 
Seligman, 1975). According to Keller (1979), learned helplessness may be “established by 
inability, impossibility of the task, or a negative set” (p. 31). Regardless of its origins, once taken 
hold, learned helplessness can be a powerful impediment to success. 

In order to help learners overcome learned helplessness and other self-fulfilling 
prophecies, it is necessary that instructional designers consider learner anxiety and provide for 
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instruction that helps boost learner eonfidenee, making them feel competent, in control, and 
successful. In general, Keller (1987a, 1987b) ealls for inereasing eonfidenee by providing for 
sueeess opportunities that are meaningful, provide adequate ehallenge, bolster achievement, and 
avoid boredom. 

It is important to note that while fear of failure can strongly affect motivation in 
traditional learning environments, it may be an even greater faetor in distanee edueation (Visser, 
1998). Even with highly-motivated students, the isolation of the learner, an unfamiliar distance 
environment, the technology required in distance eourses, the distanee separating learner and 
instruetor, and other mitigating factors have an effect on learner confidence. Studies have shown 
that teehnology brings with it new attitudes and anxiety levels that can have a direct effect on 
confidence (Yaghi & Ghaith, 2002). The instructor of the distant eourse must be especially 
concerned with increasing and maintaining learner eonfidenee. 

Method 

The purposes of this researeh were to: (a) determine if there were statistieally significant 
differences in confidence levels of online learners using systematieally designed eonfidenee 
tactics based on Keller’s ARCS model; (b) determine if the taetics also produeed a statistically 
significant difference in aeademie performanee; (e) determine if the taeties also produeed a 
statistieally significant difference in the remaining ARCS subseetions of attention, relevanee, and 
satisfaetion; and (d) determine if the taeties also produeed a statistieally significant difference in 
overall learner motivation as measured by the total ARCS score. 

The design of the instructional materials and treatment in this study was the result of 
analysis of Keller’s work on the ARCS model as well as that of numerous other researchers. 

After reviewing the literature, taeties that could be applied to improve learner eonfidenee and 
performance began to emerge. Following Keller’s (1999a, b) adviee, the eonfidenee-enhancing 
tactics were designed to be appropriate for the audience, the delivery system and the course, to 
be in-line with course objeetives and assessments, to be integrated with instruetion (provide a 
minimal level of disruption to the learning process), to be eost-effective, and to fit within the 
time restraints of the elass. 

Participants 

The subjeets in this study were undergraduate students enrolled in a for-eredit eourse at a 
Texas university rated Carnegie Doctoral/Research Universities — Extensive. Subjeets were 
seleeted from partieipants in a freshman-level eomputer eourse and were randomly assigned to 
either the treatment or the control group. 

This study was conducted over a period of approximately five and one-half weeks. The 
initial sample consisted of 81 (treatment «=41; eontrol «=40) total students and ineluded 37 
males (treatment «=18; control n=\9) and 44 females (treatment «=23; control n=2\). Ages 
ranged from 1 8 to 3 1 . Student-reported ethnicities were in-line with university-reported 
demographies eoncerning the campus undergraduate population as a whole. 

A majority of subjects (67) reported frill-time enrollment at the university. For the 
semester in whieh this study was conducted, 35 subjeets reported this section as their first 
attempt at an online class, while 44 reported having taken at least one online eourse in the past. 
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In addition, 5 subjects rated their experience and proficiency with computers as 
“beginning user.” Fifty-seven students ranked themselves as an “intermediate user.” Lastly, 15 
rated themselves as an “advanced user” with 2 self-reporting as an “expert user.” 

Research Design 

This study used a true experimental, posttest-only, control-group design, and was 
undertaken using quantitative methods (Gall, Gall, & Borg, 2003). Two quantitative surveys 
were used to measure confidence and motivation: (a) the Course Interest Survey (CIS), and (b) 
the Instructional Materials Motivation Survey (IMMS). These two surveys were delivered in 
web-based format. Performance was also measured based on the differences between posttest 
scores automatically generated in SAM Office 2003. 

This experiment used SAM Office 2003 and WebCT for the delivery and presentation of 
the tactics, strategies, confidence-enhancing e-mails (CEE), and instructional course content. 
SAM (Skill Assessment Manager) provides training scenarios for Microsoft Office in a lifelike, 
simulated environment designed to replicate Microsoft Office 2003. SAM Office 2003 was 
chosen as the delivery platform based on the prior experience of the researchers that this 
platform did an effective job of advancing student skills in Microsoft Office 2003. In the case of 
this study, the students were trained to use the Access database program. Three semesters of prior 
surveys indicated that Access was the Microsoft Office program students were least familiar with. 
Therefore, Access was chosen to help control for any variance in student ability. 

The SAM software is widely distributed to universities across the country and claims to 
have served hundreds of thousands of students and educators since its inception in 1998 (Course 
Technology Website, 2005). WebCT, along with SAM, is another widely used distance learning 
application. 

The attention, relevance, and satisfaction components of the ARCS model were not 
intentionally incorporated into the design of this study in order to better isolate the variable of 
confidence in question. 

Variables 

The independent variable (treatment) consisted of ARCS confidence tactics (see Table 1) 
distributed through SAM Office 2003 and through confidence-enhancing e-mail messages in the 
WebCT environment (see Figure 1). 

The two main dependent variables under investigation were confidence and academic 
performance. In addition, scores for the remaining ARCS components of attention, relevance, 
and satisfaction as well as an overall motivation score (ARCS total score) were calculated for 
comparison purposes. 
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CT Components Treatment Group Control 

S03: Are the tasks sequenced Materials in SAM 2003 follow a logical sequence and are The tasks were sequenced 

from simple to difficult within generally sequenced from easy to more difficult in each lesson. from simple to difficult; 
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of each exam as a percentage (e.g., 90% correct). Learners were 
reminded of this in the CEEs. 




CT Components Treatment Group Control 

PC 1 : Are learners given choices All exercises in each module were presented at once, and learners Learners were given the 
in sequencing? Can they were able to approach the lessons in any order they chose. lessons in a particular 
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Dear CECS 1100 Students, 

It is my privilege to welcome you to the fall semester of CECS 1 100. This letter serves to 
introduce myself and to offer you some advice and recommendations about the course and the assignments 
you will be completing for this class. I want you to enjoy and learn from this class, and I have no doubt that 
you will be successful! 

First, let me give you a brief personal introduction. My name is Jason Huett. I have taught this 
online section of CECS 1100 for two years now. In addition, I have taught numerous different university 
courses for the last 13 years. I am in the process of completing my PhD in Educational Computing. If you 
would like to learn more about me, please feel free to access my personal website at 
^ttp ://webpages . charter.net/xxxxxxx/ 

Second, I would like to offer some suggestions regarding how to proceed with the class. 

1 . Make sure you have logged in to the correct SAM Office 2003 section. Your section is 

2. Make sure you read and follow all the directions on your assignment sheets for each pathway. 

3. Remember, SAM Office 2003 pathways are numbered from one to eight in a clear and easy to 
follow sequence. While the assignments are available for you to complete in any order, they are 
generally sequenced from simple to more difficult. 

4. Pay attention to due dates. You have one due date for all eight Access pathways. Set a schedule for 
completion of the assignments and stick to it. Please do not procrastinate. 

5. Do as well as you can on the pretest but don’t worry if you don’t know all the answers. Feel free 
to skip any question you don’t know how to do. 

6. You can take the posttest up to six times if you desire. I will only count your highest grade. 

In these assignments, you have complete control of your pacing, your sequencing, your place of 
study, and you can have multiple attempts at the graded portion of the pathway. You will be given feedback 
regarding your performance in terms of a percent grade (e.g. 84% correct) on each exam by SAM Office 
2003. You can also access your progress report at any time by clicking on the reports button in SAM Office 
\^003. 

Lastly, I also encourage you to submit any comments you have concerning these assignments and 

how they can be improved to the section of the discussion board in WebCT. I have also set up a blog 

(short for web-log) where you can record your thoughts, comments, and ideas at http://xxxxx.blogspot.com 
Give it a try! 

Your success in this class depends entirely on you, and you will be successful 
If you ever need my help or have any questions, please do not hesitate to contact me via e-mail at 
i xxxx@xxxxxxedu . If it is an emergency, I can be reached on my cell at 940-395-3460. 

I look forward to working with you this semester. 

Sincerely, XXXXX 




Figure 1. Example of confidence-enhancing email (CEE) with comments. 



Comment [jbhl]: Confidence: 
Assure learners by showing a personal 
interest in their success. 

Comment [jbh2]: Confidence: 
Expectancy of success. 



Comment [jbh3]: Confidence: 

Personal information to build rapport and 
lower learner anxiety. 

Comment [jbh4]: Confidence: 

Learning requirements Giving students 
knowledge of what is expected of them. 

Comment [jbhs]: Confidence: 
Reminder of Success opportunities. 

Comment [jbh6]: Confidence: 
Reminder of Success opportunities 

Comment [jbh7]: Confidence: Instill 
personal control and responsibility by 
making a study plan. 

Comment [jbhS]: Confidence: 
Lowering learner anxiety. 

Comment [jbh9]: Confidence: 
Reminder of Personal Control. 

Comment [jbhio]: Confidence: 
Reminder of Personal Control 

Comment [jbhll]: Confidence: 
Reminder of Success Opportunities 

Comment [jbhl2]: Confidence: 
Reminder of Personal Control 

J 

Comment [jbhl3]: Confidence: 
Lowering learner anxiety with 
reassurance of success and reminder of 
personal control. 

Comment [jbhl4]: Confidence: 

Assure learners by showing a personal 
interest in their success. 
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Instruments 

The instruments used in the study were two surveys and series of posttests, and all were 
delivered in a web-based format at a distance. The Course Interest Survey (CIS) was designed to 
assess the four components of the ARCS model (attention, relevance, confidence, and 
satisfaction), and to provide an overall motivation score in relation to the class (Keller & 
Subhiyah, 1993). In this study, an individual measure of learner confidence was highlighted. 
Prior scores obtained with this instrument have resulted in a Cronbach’s alpha for all five 
components (attention, relevance, confidence, satisfaction and total ARCS score) in excess of .80 
(Gabrielle, 2003). For this study, scores on the web-based CIS were found to have a total 
reliability alpha of .93 based on obtained scores. The reliability alphas for the computed scores 
of the individual subsections in this study were as follows: attention (.80), relevance (.83), 
confidence (.80), and satisfaction (.83). 

Also created by Keller (1993), the Instructional Materials Motivation Survey (IMMS) 
was used to gauge the motivational effect of instructional materials. In relationship to the 
instructional material, it was designed to assess the four components of the ARCS model 
(attention, relevance, confidence, and satisfaction), as well as an overall motivation score. Again, 
an individual measure of learner confidence was highlighted. Prior scores obtained with this 
instrument (A, R, C, S, and total ARCS score) have resulted in an overall Cronbach’s alpha in 
excess of .80 (Gabrielle, 2003). The confidence subscale has previously shown a Cronbach’s 
alpha of .90 (Keller, 1993; Moller, 1993). For this study, the web-based IMMS was found to 
have a total reliability alpha of .93 based on the obtained scores. The reliability alphas for 
computed scores of the individual subsections in this study were as follows: attention (.86), 
relevance (.80), confidence (.85), and satisfaction (.86). 

Academic performance was measured using posttests automatically generated by SAM 
Office 2003 after students completed the training/instructional materials. For this study, the 
posttest was found to have a total reliability alpha of .86 based on obtained scores. 

Results 

An independent samples t-test was chosen to compare the survey responses. The 
treatment group showed statistically significant gains over the control group in terms of learner 
confidence on the Course Interest Survey (CIS) (p=.004) but not the Instructional Materials 
Motivation Survey (IMMS) (p=.080). Again, an independent samples t-test was chosen to 
compare posttest performance. In this case, the treatment group outperformed the control group 
on all of the individual posttest measures and, most importantly, on the overall aggregate mean 
performance score (p<.001; d=\). 

The results showed no statistically significant difference on the attention subsection of 
the ARCS model between the groups for either the CIS or IMMS using a conservative alpha 
measure of p=.0I. 

Statistically significant differences were noted for the relevance and satisfaction 
subscales of the model even though no intentional effort was made to enhance any variable 
except confidence. There was also a statistically significant difference in overall learner 
motivation as measured on both the CIS and IMMS. 
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Confidence and CIS and IMMS Results 

Given the design of the study, it does not unreasonable to assume the subjects receiving 
the confidence-enhancing tactics would perceive an increase in personal confidence and 
motivation for the class and the instructional materials. This assumption was only partially 
correct. While the learners in the treatment group did have higher confidence and motivation 
scores as regards the class (CIS), they did not show similar confidence gains as regards the 
instructional materials (IMMS). Quantitative results indicated there was a statistically significant 
difference between the treatment and control groups for confidence as measured by the CIS 
(p=.004). This finding was further supported by the relatively impressive effect size (J=.65) and 
estimated power (.65) at alpha .01 for the confidence subsection of the CIS. 

In addition, results indicated there was not a statistically significant difference between 
the treatment and control groups for confidence as measured by the IMMS (p=.080). However, 
the reported effect size {d=Al) and estimated power (.27) at alpha .01 cannot be dismissed as 
insignificant. Further study is warranted before a definitive conclusion can be drawn. The means, 
standard deviations, skewness and kurtosis values, effect size, and approximate power for the 
confidence variable for both measures is reported in Table 2. 

Table 2 

Results for the Confidence Subsection of the CIS and IMMS 



Subsection 


Section 


N 


Mean 


Std. 


Skewness 


Kurtosis p 


Effect 


Approx. 


Approx. 










Dev 






Size 

id) 


Power 


Power 
















(p=.05) 


(p=.01) 


CIS 


Treatment 


38 


34.79 


4.916 


-1.010 


.583 .004 


.65 


.87 


.65 


Confidence 


Control 


37 


31.46 


4.857 


-1.269 


.759 










Total 


75 


33.15 


5.135 


-.928 


.863 








IMMS 


Treatment 


35 


31.77 


7.276 


-.607 


-.207 .080 


.41 


.53 


11 


Confidence 


Control 


37 


28.70 


7.356 


-.159 


-.641 










Total 


72 


30.19 


7.428 


-.348 


-.619 
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Performance Results 

Differences in academic performance were measured between the control group and the 
treatment group using automatically generated test scores in SAM Office 2003. There were eight 
posttest subsections delivered over the treatment period. The results examined each of these 
posttest subsections individually and then as an aggregate mean score for those completing all 
eight tests. Only those completing all eight exams were included in the aggregate mean score. 
The overall average showed a statistically significant difference in performance between the 
treatment and control groups (p<.001). 

Worthy of note was the effect size of 1 for the average mean score. This can be 
interpreted to mean that the treatment group (n=30), on average, scored approximately one 
standard deviation above the average mean of the control group (n=26). Another way of looking 
at this would be to say that the mean of the treatment group was at approximately the 84th 
percentile of the control group. Though Cohen (1988) does caution against applying his 
guideline too rigidly, this most likely can be interpreted as a large effect size with substantial 
power (.91). This supports the contention that the students in the treatment group, on average, 
outperformed the control group on the posttest measures. 

Attention, Relevance, Satisfaction and Overall Motivation 

Regarding attention, at alpha .01 (p<.01) there was no statistically significant difference 
(p=.014) between the control (n=37) and treatment (n=38) groups for the attention subscale on 
the CIS. In addition, there was no statistically significant difference (p=.015) between the control 
(n=37) and treatment (n=35) groups for the attention subscale on the IMMS survey measure. 
However, the medium effect sizes (CIS d=.56; IMMS d=.51), moderate power ratings (CIS=.49; 
IMMS =.48), and the relative consistency of findings across the other subsections warrant further 
study (perhaps with larger sample sizes) before a definitive conclusion can be drawn about the 
statistical and practical implications of the applied treatment on the participants’ levels of 
attention. 

Regarding relevance, at alpha .01 (p<.01), unexpectedly, there was a statistically 
significant difference (p<.001) between the control (n=37) and treatment (n=38) groups for the 
relevance subscale on the CIS. In addition, there was a statistically significant difference 
(p=.001) between the control (n=37) and treatment (n=35) groups on the IMMS survey measure. 
The impressive effect sizes (CIS d=.S5; IMMS d=.15) and considerable power ratings (CIS=.88; 
IMMS =.80) lend credence to the claim that the treatment produced unintended effects on the 
participants’ perception of relevance for both the course (CIS) and the instructional materials 
(IMMS). 

Regarding satisfaction, unexpectedly, there was a statistically significant difference 
(p=.001) between the control (n=37) and treatment (n=38) groups for the satisfaction subscale on 
the CIS. In addition, there was a statistically significant difference (p=.002) between the control 
(n=37) and treatment (n=35) groups on the IMMS survey measure. Similar to relevance, the 
impressive effect sizes (CIS <7=.78; IMMS d=.72) and considerable power ratings (CIS=.83; 
IMMS =.73) lend credence to the claim that the treatment produced unintended effects on the 
participants’ perception of satisfaction for both the course (CIS) and the instructional materials 
(IMMS). 
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Discussion 



Based on the study’s design, one could potentially expect the results to reflect an increase 
in learner confidence on both the IMMS and CIS measures. In fact, given that many of the 
confidence tactics used in this study were specifically designed to enhance the instructional 
materials, one would expect the IMMS to show a noticeable increase in learner confidence even 
if the CIS did not. However, quite the opposite was the case: the students in the treatment group 
found the tactics confidence-enhancing as regards the class as a whole but not the instructional 
materials. In a similar study, Moller (1993) failed to obtain changes in confidence as regards 
instructional materials and listed three possible explanations: 

1) The ARCS model is insufficient for improving learner confidence; 2) the resulting 
tactics and methods used in the research were inappropriate for these subjects or 
implemented improperly; and 3) the differences were too small to measure using the 
selected empirical research methods, (p. 89) 

To this and echoing Babe (1995), the researchers would add: 4) perhaps the role of the 
confidence variable, as one of the four main subsections of motivation, needs to be reexamined. 

Taking each of these possible explanations in turn, there is insufficient data to suggest 
that the ARCS model is somehow flawed or incomplete when it comes to addressing learner 
confidence. The model has shown an ability to increase learner confidence even when 
confidence was not the focus of the researchers’ investigations. The fact that the model, as a 
whole, can produce increases in confidence is not really in question. The question is more 
whether the individual subsection of confidence can be targeted as a valid, independent construct 
that produces consistent results. 

Along with other research, this study’s rather mixed results suggest that confidence may 
indeed be a more abstract and complicated dimension in the overall realm of motivation than the 
ARCS model would lead one to believe. Researchers who have specifically targeted confidence 
for enhancement have been unable to validate the independence nature of the confidence 
component under controlled conditions (Moller, 1993; Naime-Diefenbach, 1991). As mentioned 
earlier, researchers such as Gabrielle (2003) and Babe (1995) found mixed returns for confidence 
even when confidence was not the focus of the study. 

Given the complexity of isolating confidence and of dealing with the cognitive and 
affective domains of the individual as well as concepts such as anxiety, locus of control, and fear 
of failure, it appears one weakness of the ARCS model may be its oversimplification of the 
abstract and highly complex concept of confidence. In order to obtain an increase in confidence, 
it may be necessary to take a more comprehensive approach to motivating students and include 
enhancements to the other ARCS components even if the desire is to focus on confidence alone. 
This is supported by researchers such as Marovitz and Buckley (1987), who felt the results of 
their experiments indicated “that the four factors of Keller’s ARCS model are intricately bound 
together” (p. 12). This would require a rethinking of the ARCS model as a series of related and 
not independent constructs for improving motivation (Babe, 1995). 

The second possible explanation for the lack of a statistically significant difference in 
confidence regarding the instructional materials is that the confidence tactics and confidence- 
enhancing e-mails used in this study were ineffective or implemented improperly. This is always 
a possibility. A few strategies that may have had an effect on confidence were “built-in” to the 
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SAM Office 2003 software (such as any novelty effects generated through the simulation 
program or student choice over study location) and could not be removed. This may have 
impacted the confidence levels of participants in unforeseeable ways. In addition, treatment 
group subjects indicated that they found some of the confidence tactics used in this study more 
effective than others. 

For example, informal surveys of participants indicated they found the guide-sheets for 
each Access assignment and the e-mail messages, which incorporated all of the tactics (see Table 
1) under Component I: Learning Requirements and most of those under Component II: Success 
Opportunities, as confidence-boosting. However, only two students took advantage of the 
opportunity, under Component III: Personal Control, to create their own exercises or methods of 
demonstrating competency. Also, under the same component of personal control, only two 
students accessed the blog and threaded discussion on how to make the materials more 
interesting. Finally, even though students were given a choice to control their own sequencing, 
almost all chose to complete the assignments in the same order. 

Researchers have linked increases in learner control to increases in confidence (and 
positive attitudes of learners) as well as decreases in learner anxiety (Bandura, 1977; Keller & 
Suzuki, 1988; Kinzie, 1990; Kinzie & Sullivan, 1989; Moller & Russell, 1994). However, in this 
study, some members of the control group indicated an appreciation for the strict structure, 
deadlines and pacing. A majority of treatment group students (64%), who were given personal 
control to complete the assignments at any time during a five-and-one-half-week window, waited 
until the last 72 hours to “cram in” most of the assignments before they were due. Only 24% of 
the treatment group finished the required assignments before the last week. This is not altogether 
surprising since, according to Ferrari, Keane, Wolfe, and Beck (1998), “as many as 70% of 
American college students engage in frequent academic procrastination” (p. 199). 

There is no real way of knowing how such procrastination affected the confidence levels 
of the treatment group, but one can imagine that procrastination brings with it an increase in 
learner anxiety. Anxiety has an inverse relationship to confidence, so the effect on confidence 
levels was probably not a positive one. Wolters (2003) cited more than a dozen studies linking 
procrastination to higher levels of anxiety and lower levels of self-esteem. Milgram, 

Marshevsky, and Sadeh (1995) found that students with fewer abilities to manage their own 
learning requirements tended to procrastinate more when given a choice over when to begin their 
own tasks. This could account for the lack of difference in confidence as measured on the IMMS. 
If students are waiting to the last minute to complete the materials, the treatment does not have 
very long to take effect. 

One may need to reexamine certain aspects of personal control (including the tactics used 
in this study) to determine if they are indeed confidence-enhancing for subjects similar to those 
in this experiment. One may also want to examine in more detail how personal control issues 
affect distance learning environments in particular. What seems clear is that the students in this 
study did not use some of the tactics as intended. In fact, some of the tactics used in this study 
(mostly those under Personal Control) may have had the opposite effect of what was intended. 
Even Keller admits that allowing learners “to control the instructional strategy of a lesson may 
not be beneficial” (Klein and Keller, 1990, p. 145). Such findings further the idea that trying to 
isolate confidence when dealing with diverse groups of individuals, with differing levels of 
maturity, may be a more difficult process than first envisioned. 
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Third, another way of stating differences may be too small to measure is to say perhaps 
the IMMS survey is not sensitive enough to detect short-term changes. While no current research 
indicates that the IMMS survey is a poor or weak measure of learner confidence, it seems 
possible that this survey may not be sensitive enough to detect short-term changes. Perhaps the 
confidence enhancements are producing a desired effect, but the survey cannot consistently 
detect the changes over the short-term. Given the protean motivational nature of learners over 
time, the survey would need to be highly sensitive and/or delivered at precisely the right time to 
accurately reflect learner changes in confidence in the short-term. Over a brief period, learners 
may not even be aware enough of a change to report it accurately. 

The question of study duration brings up other issues. Moller (1993) writes “assuming the 
longer an attitude is held the stronger it becomes, it may be unrealistic to assume that a 
measurable change [in confidence] can be detected using a short-term experimental design” (p. 
92). The meaning of “short-term” is unclear. A wider review of the literature does little to clarify 
exactly how much time it takes for the ARCS model effect a statistically significant difference in 
learner confidence. 

Looking at previous studies, one finds a diverse spectrum of study durations resulting in 
different assessments of subjects’ confidence on the IMMS. For example, Moller (1993) and 
Naime-Diefenbach (1991) studied confidence using written instructional materials in one short, 
self-instructional lesson and showed no noticeable changes in confidence as measured on the 
IMMS. Babe (1995) used a longer instructional lesson with relevance-enhancing strategies and 
showed an increase in learner confidence on IMMS. This study used a five-and-one-half-week 
treatment period with no confidence changes on the IMMS but notable changes on the CIS. 
Gabrielle (2003) applied her treatments to a highly homogenous group of military cadets over 
one long semester and showed increases in confidence on the IMMS but not the CIS. In short, 
there is no clear picture of how much time is necessary to identify noticeable confidence changes 
for either the instructional materials or the class as a whole. 

Lastly, there is the idea that the confidence subsection should be reexamined as one of the 
four main constructs of the ARCS model of motivation. With such issues as maturity, anxiety, 
locus of control, and fear of failure, confidence may not lend itself to easy encapsulation inside 
of a model. There is no doubt that confidence is a part of motivation, but the ARCS model 
implies that there are specific strategies and variables related only to confidence. These strategies 
for improving confidence are independent of the other ARCS components and, hence, can be 
individually targeted and manipulated (Keller, 1987a). 

Naime-Diefenbach (1991) claimed to have validated the independent attention 
component of the ARCS model. Babe (1995), Nwagbara (1993), and Chang (2001) validated the 
usefulness of targeting the relevance component, but Babe questioned its independent nature. 

The researchers could locate no studies specifically targeting the satisfaction component, but one 
can speculate that targeting instruction for increases in student satisfaction would not possess the 
same degree of complexity as enhancing learner confidence. 

To the researchers’ knowledge, all studies to date, including this one, that have used the 
ARCS model to specifically target confidence have failed to achieve statistically significant 
results as reported by the IMMS. Possibly, placing the more abstract and difficult dimension of 
confidence on equal footing with the other components of the model diminishes the conceptual 
validity of the model. 
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Performance 

The data showed that the students in the treatment group, on average, outperformed the 
control group on the posttest measures. This is in-line with previous research findings that 
suggest increases in motivation can translate into increases in performance or achievement 
(Bickford, 1989; Gabrielle, 2003; Song & Keller, 2001). 

Since the treatment group was exposed to a pretest exercise as a confidence tactic, some 
of the difference in scores may have been attributable to a “practice effect.” However, it is the 
researchers’ contention that this effect would not account for all the variance in scores. 

A, R, S and Overall Motivation 

For this study, there were clear differences noted for relevance, satisfaction and overall 
motivation. By way of comparison to other studies, Moller (1993) showed no changes in any of 
the ARCS subsections using the IMMS. Targeting overall motivation, Gabrielle (2003) found 
statistically significant differences only for attention with a moderate difference for satisfaction 
(p=.076) and no difference on the relevance and confidence subsections of the CIS. Regarding 
the IMMS, she found statistically significant differences on all subsections. In a study that 
specifically manipulated the attention and confidence subsections, Naime-Diefenbach (1991) 
showed a statistically significant increase in attention but no increases in confidence or the 
remaining ARCS subsections of the IMMS. As one can see, consistency of findings is an issue 
across studies. 

One particular reason for the mixed findings in this study may be an overlap of the 
confidence tactics (see Table 1) and confidence-enhancing e-mails (see Figure 1) into the 
attention, relevance and satisfaction components. For instance, providing the treatment group the 
opportunity to create their own exercises or methods of demonstrating competency (PC3) and 
allowing the treatment group access to a blog and threaded discussion for comments (PCS) may 
have enhanced attention or even relevance. Tactics such as these might stimulate the learner’s 
curiosity to think of ideas for improvement that increase feelings of “connectedness,” or 
relevance, to the material. Allowing learners multiple entry points into the instruction (SOI), 
which catered to individual skills and avoided wasting time, might have increased the learner’s 
sense of satisfaction as well as confidence. 

Although the confidence-enhancing e-mails used in this study were designed to stress 
only the confidence tactics and strategies, it is possible that they had an indirect effect on the 
treatment group’s sense of attention, relevance, and satisfaction. Simply receiving the e-mails 
might serve to gain learner attention. The concern, verbal praise, and goal reminders expressed 
in the messages may have served to increase learner satisfaction and improved their sense of 
connectedness (relevance) to the subject matter. Also, SAM Office 2003 is a simulation program, 
and simulations and real-world settings are suggested by Keller for enhancing both satisfaction 
and relevance (Babe, 1995). However, this study’s findings still call into question the 
discriminate validity of the separate categories of the ARCS model. At the very least, this study 
highlights the challenges faced in isolating confidence for independent enhancement. 

Perhaps the most important finding is that overall motivation can be enhanced in learners 
through the application of external factors. That was the belief which initially guided this study 
and, despite any disagreement about the validity of the independent components of the ARCS 
model, the model, as a whole, proves to be an effective design tool for increasing overall learner 
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motivation. The findings of this study confirm decades of previous research that motivation is a 
critical component to learning (Keller, 1979, 1987a, 1987b, 1987c; Means, Jonassen & Dwyer, 
1997; Song & Keller, 2001). This study also furthers the body of research by affirming the 
ARCS model as a viable model for increasing motivation and performance in distance education 
settings. 

Implications for Instructional Design 

The results of this study offer several suggestions for future researchers and instructional 
designers. First, overall motivation can be enhanced in distance learners through the application 
of carefully crafted external factors such as confidence tactics and confidence-enhancing e-mails. 
Second, the performance results of this study show that motivation is a powerful force in 
learning. This study confirms that systematically designed and carefully applied tactics can 
improve performance. Third, Keller’s ARCS model is an effective design tool for building 
motivational enhancements and e-mails into distance education environments, and one should 
not shy away from using Keller’s ARCS model as a conceptual framework for new and 
emerging technologies. If one believes that distance learning environments pose greater 
challenges to learner motivation than their face-to-face counterparts, then a well-thought-out 
systematic approach to manipulating distance learner motivation is an important design 
consideration. 

While the study did not support the discriminate validity of the separate category of 
confidence, it does provide future designers with material proven useful for increasing overall 
motivation and performance. One implication future designers may wish to take from this study 
is to focus less on individual aspects of the ARCS model and more on a learner’s overall sense of 
motivation. To this end, confidence is a powerful variable that needs to be included in 
instructional design. 

Further Research 

The results of this study have spawned several ongoing research projects. Many of the 
confidence-enhancing tactics used in this study were delivered through e-mail. This has led to 
follow-up studies focusing on motivational messages similar to the one illustrated in Figure 1. 

Following the work of researchers who found positive outcomes in learner motivation 
and/or retention using motivational messages (Gabrielle, 2003; Huett, 2006; J. Visser, 1990; 
Visser and Keller, 1990; Visser, Plomp, and Kuiper, 1999; 2002), the ARCS model is being used 
as a guide for developing motivational communications with distance education students. In one 
ongoing study conducted by the authors, the initial results look promising. Though data analysis 
is still underway, in two large distance education computer classes (where the only difference is 
the presence of ARCS-based motivational e-mail messages delivered at roughly 10-day intervals 
for the treatment), preliminary results show significant increases in student motivation, 
satisfaction, relevance, and attention. Similar to this study, no significant differences have been 
noted for confidence. Also, the treatment group shows a 6.35% failure rate compared to that of 
19% for the control. In addition, the treatment group has an average drop-out (non-completer) 
rate of 4.76% compared to 15.5% for the control. 

Other studies using similar messages are currently in the early stages of replication with 
distant students, from undergraduates to graduate, in online classes dealing with a variety of 
subject matters. It seems feasible that the social aspect or sense of community created through 
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motivational communications may be part of the necessary support structure distance learners 
need (Cathcart, Samovar, & Henman, 1996; Kember, Lai, Murphy, Shaw & Yuen, 1994; Moller, 
1998). 

While additional research is needed to validate the effects of these motivational 
communications, ongoing research suggests that efforts to improve motivational communications 
in distance learning situations could have significant returns. 
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Abstract 

This study adopted developmental research methodology to systematically investigate the process of applying 
instructional design, human-computer interaction, and software engineering principles to a Performance Support 
System (PSS). To facilitate this investigation, Ausubel’s advance organizer theoretical approach was used to 
integrate the instructional design principles into the PSS; a grid interface with matrix-based design metaphor 
was incorporated to facilitate human- computer interaction between users and the PSS; and participatory rapid 
prototyping was utilized to engineer the PSS. Developmental data were collected from a six-stage participatory 
rapid prototyping process. Findings suggest that the proposed advance organizer concept combined with the 
matrix design metaphor has the potential to foster participants’ metacognitive awareness in the selection of 
possible problem solutions and to enhance participants’ searches for relevant content information. 

Introduction 

Developmental research, “the systematic study of designing, developing and evaluating instructional 
programs, processes, and products that must meet the criteria of internal consistency and effectiveness” (Richey, 
1994, p. 714), is one of the research methods used in instructional development where the information produced 
by the research is mainly a resource to inform instructional designers of how particular problems have been 
resolved. By conducting developmental research, instructional researchers are able to explore and document the 
instructional procedures used by the developers - how they identify admissible alternatives and then proceed to 
make decisions about the most satisfactory among them. Sanders (1981) first used the term “developmental 
research,” proposing it as a better way to understand the instructional process than were conventional 
laboratory-based approaches. Whereas laboratory-based research attempts to describe a phenomenon through a 
well-controlled experimental setting, and naturalistic research attempts to generalize from a phenomenon, 
developmental research is a process "directed at determining whether, through particular interventions, desirable 
results can be achieved, and if so, under what conditions" (Sanders, 1981, p.l 1). By “interventions,” Sanders 
means the process of continual refinement by which a developer improves a system with feedback from its 
users. The advantage of developmental research is that it is “applicable to those creative, indeterminate 
processes that are intractable through conventional means” (Sanders, 1981, p.l 2). 

There are several good reasons for using developmental research to understand instructional 
development and other education-related processes. Perhaps the main purpose of developmental research is “the 
invention and improvement of creative approaches to enhancing human communication, learning, and 
performance through the use of technology and theory” (Reeves, 1996, p. 462). Many researchers have 
proposed or conducted developmental research during the creation of instructional programs and learning 
environments to advance the practice of instructional development (Clark, 1989; Terlouw, 1993; Reeves, 1996; 
Reigeluth & Frick, 1999; van den Akker, 1999; Richey, Klein, & Nelson, 2004). Studying, describing, 
analyzing, and reporting the design and development process of an innovative instructional program or product 
help us to better understand how theoretical frameworks (e.g., learning theories and instructional principles) are 
applied to that process and how we should revise both the program and its design framework. 

The present study sought to investigate the issues involved in the development of a Performance 
Support System (PSS) designed for classroom behavior management by teachers. The goal was to understand 
how instructional design principles, human-computer interaction, and software engineering can be applied to the 
design and development processes to create an “instructionally integrated” PSS. With this integrative approach, 
this study examines how the proposed design framework and development process can (a) transform the PSS 
technology into an instructionally integrated approach.. 
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The Inherent Problems of Technically-Driven PSSs 



Although the design framework of a performanee support system (PSS) has not been elearly defined in 
the extant literature, the major teehnieal eharaeteristie of sueh systems has been identified as a module-based 
approaeh whieh allows for flexibility in the exehange of modules to meet users’ needs and provides the 
possibility of parallel development for eaeh funetional eomponent (Barker & Banerji, 1995; Gery et ah, 2000; 
Raybould, 1995; Tjahjono & Greenough, 2002; van Sehaik et al., 2002). Based on this teehnieal eharaeteristie, a 
PSS offers designers a way to integrate multiple instruetional modules (sueh as a tutorial, hypertext, advisory 
systems, information management systems, and eollaboration tools) into a paekage with a eommon interfaee 
that provides users with a helpful system for dealing with various tasks and problems eneountered in their work 
environment. However, sueh a generie design approaeh is potentially problematie beeause it relies eompletely 
on teehnology to “glue” all the modules into one unit with an interfaee as the wrap. 

By integrating the modules through a eommon interfaee, the PSS is merely “teehnieally integrated,” 
but not “instruetionally integrated.” That is, eaeh module is linked and eonneeted by the interfaee alone. Sueh an 
interfaee, if appropriately designed, may help users navigate among modules with ease (e.g., via effeetive 
arrangement of sereen elements) and provide improved orientation to the system’s teehnieal strueture (e.g., 
through uniformity & eonsistent sereen layouts), yet it may not be enough to support eontent eomprehension and 
problem solving aetivities that are inherent in the learning and performanee goals of a PSS. This is beeause users 
are still working with separate system eomponents without additional guidanee or seaffolding to lead them to 
other relevant modules. Therefore, in order to support eontent and learning aetivities, an effeetive PSS interfaee 
should ineorporate instruetional attributes (e.g., eoneept maps, advanee organizers) to help users anehor their 
knowledge and skills aequisitions in meaningful eontexts. 

In addition, beeause traditionally eaeh PSS module is funetionally independent, the proeess eould result 
in a “shotgun” style design; the designer hopes that at least one of the many modules will be usable in order for 
the whole system to be useful. This is a passive design approaeh that does not provide struetures and 
relationships among eomponents to allow users to take full advantage of the aetivities and support eontained in 
eaeh eomponent. Sueh a design is also heavily teehnology-driven, and designers (and developers) ultimately 
may be preoeeupied with teehnieal problems, rather than with supporting users. Therefore, to overeome the 
problems eaused by a generie, teehnology-oriented approaeh, a PSS design should begin with the integration of 
an instruetional design framework to inerease its pedagogieal usability. This approaeh ean also help organize, 
strueture, and evaluate relevant teehnieal eomponents for on-the-job task learning and performanee gains. 

Proposed Design Frameworks 

In order to design a PSS that is integrative (rather than “shotgun” oriented), performanee-eentered, and 
instruetionally purposive, the present study ineorporated features of instruetional design prineiples, human 
eomputer interaetion, and software engineering to guide the seleetion and integration of system modules. In this 
study, we proposed three design strategies to aehieve this integrated framework: (a) advanee organizer-based 
instruetional design prineiples for seaffolding, (b) a matrix metaphor to support and guide users’ eontent 
retrieval and aeeess to PSS modules, and (e) a grid-based interfaee design to integrate the PSS modules. The 
relationships among the integration strategies, theoretieal support, and design proeedures used in this study are 
outlined in Table 1. These three design strategies were then operationalized through a partieipatory rapid 
prototyping development proeess. 



Table 1: Proposed Design Strategies 



Integration strategies 


Purpose (Theoretieal Support) 


Method (Design Proeedures) 


Seaffold users’ 
eontent 

eomprehension and 
problem analysis 
through advanee 
organizers 


1) Guide the seleetion of PSS 
modules (together with the matrix 
design strategy) 

2) Meet expert/user expeetations 

3) Raise users’ eognitive awareness 
of the unfamiliar domain knowledge 
that will be leamed/performed 


1) Begin with a eomparative organizer to 
provide ideational seaffolding for the eontent 
materials to be learned. 

2) Add expository organizer to provide 
inelusive information based on the information 
residing in the eomparative organizer. 

3) Ineorporate expert/empirieal based strategies 
and aetions to operationalize information 
learned from both organizers. 


Guide and lead 
users’ eontent 
retrieval and aeeess 
to PSS modules 
through a matrix 


1) Guide the seleetion of PSS 
modules (together with the advanee 
organizer design strategy) 

2) Transform the eoneeptual 
framework of expository and 


1) Use a single sereen interaetion approaeh to 
improve eomprehension of eontent presentation 
and avoid diseonneeting of eontent relevanee 
and navigation loss (Hammond, 1993). 

2) Strueture information in three ways to 
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metaphor 


eomparative organizers into a work 
panel for problem identifieation 
3) Provide visual overview of eontent 
materials to remind and give hints to 
learners about subsequent tasks to be 
performed 


inerease the learning materials’ applieation and 
eontext relevanee. 

3) Use hypertext teehnology to give users 
aeeess to more information on that eore eontent 
as neeessary for eomprehension. 


Visually unite the 
seleeted PSS 
modules with a grid- 
based interfaee 


1) Create an identieal interfaee that 
would unite system modules to ensure 
eonsisteney in their looks and feels 

2) Minimize the extent to whieh 
users need to be aware of what they 
are doing. 

3) Inerease users’ knowledge of how 
to use available funetions, rather than 
inereasing their knowledge about 
some speeifie aspeet of the strueture 
of these funetions 


1) Develop a set of shared elements (e.g., print, 
bookmark, note taking, glossary) to maintain 
eonsisteney in the interfaee ’s navigation 
strueture and orientation 

2) Employ a grid design teehnique along with 
the “golden ratio” aesthetie prineiple to 
proportion the navigation strueture and eontent 
layout. 

3) Divide the entire window into bloeks of 
equal size and then start to assign a eertain 
number of bloeks to eaeh area. 



Our goal for applying an advance organizer to the system interfaee was to make the overall eontent 
information readily available to the users. This would allow users to mentally eonstruet an exploration and 
eomparison proeess and faeilitate development of the skills required for problem solving. To aeeomplish this, 
we ereated a multi-dimensional matrix strueture that was eombined with a funetional and visual design 
metaphor to graphieally lay out the planned instruetional strategies on the system work areas. Graphically laid 
out means that the instruetional design strategies (i.e., expository and eomparative) were transformed into four 
visual areas. Speeifieally, these are areas for (a) eomparing teaehers’ existing knowledge with new knowledge; 
(b) exploring inelusive information about the new knowledge; (e) exploring applieations (i.e., interventions) of 
inelusive information; and (d) applieation strategies for the inelusive information. These four visual areas were 
then integrated and sequeneed into the multi-dimensional matrix strueture. Within this matrix strueture, 
presentation of the inelusive information of the eontent materials is represented in a single page (or single 
eomputer sereen) to provide teaehers with a holistie view of the eontent materials as well as their inter- 
relationships, thereby providing a potentially meaningful learning environment. Figure 1 shows how these four 
visual areas are integrated and sequeneed in the multi- dimensional matrix strueture to ereate the Behavior 
Matrix module. 



Figure 1 : Sereen shot of the matrix-based behavior information organizer 




c 



d 



System Developmental Research 

The purpose of the present study was to examine how teaehers pereeive, reaet, and use the above- 
deseribed design strategies in an aetual work setting during the system development proeess. In this sense, the 
study was developmental researeh intended to systematieally explore the eonsisteney and effeetiveness of an 
instruetional produet’s design, development, and evaluation proeess (Riehey, 1994). From the study of this 
systematie proeess, we sought to gain a detailed understanding of how the design affeets users, as well as issues 
that arose during the development proeess. The following developmental researeh questions were examined in 
this study: 

1 . How are design strategies realized and what issues are eneountered in the development proeess? This 
researeh question was adopted to guide our researeh in the design and development proeesses. 
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2. How do the design strategies (advanee organizer, matrix strueture, grid-based interfaee) affeet the 
teaehers’ development of intervention strategies? This researeh question was adopted to guide our researeh 
in the evaluation proeess. 

Project Participants 

Thirteen partieipants from a private foundation’s existing 28-member foeus group were reeruited after 
they indieated a willingness to partieipate in this study. All partieipants were either elementary or junior-high 
sehool teaehers from three sehool distriets in the suburban Chieago area and most of them had more than ten 
years of teaehing experienee. Among the thirteen partieipants, three were resouree teaehers and ten were speeial 
edueation teaehers. Partieipants were asked to eomplete a ten-month, six-stage system development evaluation 
proeess, ineluding assessment, design, prototyping, planning, aetion, and audit. 

In addition to the thirteen partieipants, two university professors from instruetional development and 
speeial edueation respeetively, a pediatrie doetor who speeialized in attention- defieit disorder and prenatal drug- 
exposure, a retired speeial edueation teaeher, a elinieal psyehologist, and a software engineer who speeialized in 
relational database development also partieipated as subjeet matter experts (SMEs) in expert appraisals during 
the development of the system design. Additionally, a projeet design and development team of five persons was 
formed, eonsisting of three eontent experts, one edueational software developer (who also was the primary 
researeher), and one media developer. 

Research Methods 

A mixed methodology was used in this study to address the eomplexity of the development proeess. 
Eaeh development stage adopted different data eolleetion methods to meet its unique needs. These data 
eolleetion methods ineluded foeus groups, expert appraisal, usability evaluation, user interviews, impaet survey, 
and a ease study. Data eolleetion instruments ineluded a user profile questionnaire, system appraisal eheeklist, 
evaluation tasks for usability testing, a field interview guide, and an impaet questionnaire. Detailed deseriptions 
of eaeh data eolleetion method and instrument are organized aeeording to the development stages and presented 
in the findings seetion. 

Research Procedure: System Development Approach 

To apply the three design strategies in aetual development, we adopted a rapid prototyping proeess 
(e.g., Jones and Riehey, 2000; Tripp and Biehelmeyer, 1990) to develop the PSS system prototype. Rapid 
prototyping is a method eommonly used in software engineering. Its purpose is to bring projeet sponsors, users, 
eontent experts, and a development team together to diseover effeetive solutions to dynamie problems through a 
seamless iterative proeess of designing, eonstrueting, and evaluating prototypes (Pressman, 1992). When 
developing the system prototype, rather than minimizing the task by merely developing a mock-up version of the 
system, we attempted to explore any possible tasks involved in an aetual setting by engaging in a small-scale 
version of the design and development proeess. Thus, the prototyping eould be applied within a short eyele of 
time to portray the eharaeteristies of the full-seale version (Tripp & Biehelmeyer, 1990). 

A eommon shorteoming of previous studies using rapid prototyping has been the laek of speeifie steps, 
rules, and guidelines (Dey, Abowd, & Salber, 2001; Lohr, et al., 2001; Nixon & Lee, 2001). To address this 
potential shorteoming, we added a systematie approaeh, the participatory design process model developed by 
Grantham and Niehols (1993), as shown in Figure 2, to help us operationalize the system development, data 
eolleetion, and evaluation in a eontinuous proeess. In this looping design approaeh, the key to sueeess involves 
following the sequenee in order (1-2-3-4-5-6) and repeating the proeess (without omitting any steps), until the 
desired results are aehieved. Users need to be eontinuously involved throughout the design eyele. By applying 
partieipatory design teehniques within the rapid prototyping proeess, we sought to maintain the development 
proeess in a struetured, eonsistent, and systematie manner. 

Figure 2. Six-Stage Partieipatory Design Proeess (Derived from Grantham & Niehols, 1992) 

6. System Audit 1 • Assessment 




4. Planning 3. Prototype 



Findings and implications of the system development process 

In this seetion, data eolleeted from eaeh stage are reported and our refleetions about the development 
proeess are also deseribed to better illustrate the nature of real-world instruetional system development. 
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Stage 1: Assessment. The purpose of this stage was to assess; (a) how teaehers develop intervention strategies 
for elassroom behavior management; (b) what skills and knowledge need to be aequired to beeome a profieient 
problem-solver when dealing with elassroom management issues; and (e) how PSS funetionality eould be 
ineorporated to support learning and on- task performanee. Results of this analysis were used to help us exeeute 
both design strategies (development of a matrix-based advanee organizer and seleetion of appropriate PSS 
modules). Teaeher foeus group interviews were eondueted to solieit information on what system modules ean 
and should be ineorporated to the present system. Prior to the teaeher foeus group, a user profile survey was 
administered to gather data eoneeming eaeh partieipant’s eomputer usage and experienee with elassroom 
behavior management. Findings from the survey helped the projeet team plan the foeus group interviews and 
generate a list of system funetions that eould be ineorporated to the system. 

Data from the profile survey indieated that the mean teaehing experienee of the partieipants was ten 
years, and they all reeeived at least two training workshops for elassroom teehnology integration. Partieipants 
also reported that they spent, on average 2-3 hours daily on the eomputer engaging in the three most eommon 
aetivities: (1) word proeessing, (2) Internet searehes for teaehing resourees, and (3) email aetivity. Furthermore, 
when asked about their experienee with elassroom behavior management, most partieipants reported that they 
rely on their teaehing experienee and information given by the sehool resourees teaehers and students' parents to 
develop interventions for elassroom behavior management. They seldom relied on eomputer (e.g., searehing the 
Internet for relevant resourees) and book referenees to assist in their development of elassroom intervention 
strategies. It was evident that the partieipants were expeeted to aet independently to resolve their daily 
elassroom behavior issues. 

Partieipants’ experienees with elassroom behavior management was further addressed in the foeus 
group interviews to determine how the proposed system eould further support teaehers’ individual performanee 
in resolving their daily elassroom behavior issues. Information obtained from these foeus groups suggested that 
this phenomena was three- fold. First, the problems that partieipants eneountered often required immediate 
attention. Seeond, the partieipants were aeeustomed to solving these problems on their own or with parents of 
the students, unless the problems were beyond their eapability, in whieh ease they would turn to the resouree 
teaehers. Third, the availability of resouree teaehers, the laek of explieit guidelines about elassroom behavior 
management, and the large number of students in the elassroom eneouraged the partieipants to plan their own 
interventions without asking for help. 

Partieipants also indieated that a meehanism for managing their development of intervention strategies 
and keeping traek of the implemented interventions would in faet help them doeument their efforts in elassroom 
management and enhanee their ability to deliver better instruetion in the elassroom. Sueh meehanism would 
partieularly help partieipants when eompleting an Individual Edueation Plan (lEP) whieh requires speeial 
edueation teaehers to keep traek of individual student’s progress. 

From these findings, it was eonfirmed that a system that foeused on individual performanee and just-in- 
time support was needed. As a result, when the partieipants were asked to ehoose a list of twelve possible 
system modules that they would like to have ineorporated into the performanee support system, the majority 
agreed that a feature-rieh eontent knowledge base would be most benefieial to them in finding relevant and 
usable information for their development of intervention strategies. Next, a system module for them to apply? 
the knowledge that they have aequired and a system module for eollaboration opportunity among resourees 
teaehers and parents were also favored by the majority of partieipants. It is important to note that partieipants 
had mixed feelings about a reeord keeper type of system module. Partieipants thought that the reeord keeper was 
an important funetion to be ineluded in the future PSS beeause it would help them to organize and reeord the 
intervention strategies that they use with students. However, other partieipants questioned the usefulness of this 
module in the field beeause they already have data bases to perform similar funetions and thus may not want to 
adopt a new one. Furthermore, partieipants pointed out that some of the sehool distriets use standard data bases 
to reeord students' performanee and behavior. These data bases are typieally eonsidered to be sehool property 
and eannot be replaeed or transferred. Finally, the diseussion eentered on how information management should 
be redesigned so that it would be more suitable for teaehers in general. 

In response to this feedbaek, the projeet team struetured the system design framework around the 
support of individual performanee needs - by providing a set of performanee and referenee tools that eould help 
the partieipants address their immediate needs, while also offering opportunities for knowledge advaneement. In 
addition, eommunieation features were planned so that users would have a eollaboration funetion that faeilitated 
partieipation in a diseussion group or faeilitator-monitored eollaborative learning. As a result, four additional 
system modules (in addition to the behavior matrix) were seleeted and adopted. The seleetion of the four 
modules was determined by: (1) how aeeeptable the system modules would be to the partieipants; and (2) how 
the system modules eould be ineorporated to support general problem solving approaehes used in the field of 
elinieal psyehology. 
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Stage 2: Design. After the seleetion of system modules was made, a fully interaetive storyboard of the system 
interfaee and navigation strueture was ereated based on the proposed matrix metaphor and grid based interfaee. 
The goal was to represent the visible eontents of eaeh sereen (e.g., graphie, text, title, video) and system 
interaetivity (e.g., behavior matrix interaetion) that would be eneountered by the users. Mierosoft PowerPoint 
was used to assemble the eontent and flow of every sereen the users eould eneounter. This storyboard was then 
reviewed by the panel of subjeet matter experts (SME). The six SMEs were invited to partieipate in two Delphi 
sessions of expert appraisal to elieit their opinions on the design strueture and system module integration. The 
Delphi sessions followed the proeedures suggested by Vennix and Gubbels (1994). The first Delphi session was 
eondueted as a one-on-one appraisal session to elieit eaeh SME’s opinions and understanding of the design 
strueture and system module integration. The seeond Delphi session ineluded all six SMEs, who partieipated in 
a two-hour group diseussion to aseertain the praetieality and versatility of the design strueture and system 
modules. 

In the first Delphi session, after the presentation of the storyboard, a system appraisal eheeklist was 
distributed to eaeh SME. The list eontained 75 yes/no questions that were divided into three main eonstruets: 
instruetional features, interfaee & orientation, and teehnieal aeeuraey. Data gathered from the individual 
eheeklist were eolleeted and summarized. Twelve items were identified that refleeted less than 90% eonsensus 
(i.e., fewer than 5 out of 6 SMEs eheeked the same answer). These items ineluded five items from the 
instruetional features eonstruet eoneeming module and topieal sequenee, navigational depth of behavior 
management eontent and user data reeord eoverage; four items from the interfaee & orientation eonstruet 
regarding ieon appearanee, size of eontrol, and reversal of undesirable aetion; and three items from the teehnieal 
aeeuraey -eonstruet regarding feedbaek of task eompletion and labeling. These 12 items were redistributed in a 
seeond Delphi session for group diseussion and to elieit suggestions for improvement. 

Before administering the seeond Delphi session, the projeet team made a minor revision to the 
storyboard to address a number of immediately obvious eoneems and issues, sueh as the size of navigational 
ieons and headings listed in the 12 non-eonsensus items. This revised storyboard, along with the 12 non- 
eonsensus items were then brought to the seeond Delphi session where all SMEs met and diseussed the 
possibilities for enhaneement and improvement. In this session, eaeh of the 12 items was presented to the SMES 
to elieit suggestions and eomments on improvements. The projeet team eolleeted the feedbaek and then 
attempted to eolleetively reaeh an overall revision plan for eaeh non-eonsensus item. For example, when 
diseussing the navigational depth of behavior management eontent, both the instruetional teehnology professor 
and elinieal psyehologist felt there were too many layers of matrix eontent and the size of pop up windows was 
too small. In addition, the speeial edueation teaeher felt the eontents were unorganized and diffleult to follow. 

To respond to these eoneems, the projeet team would make direet adjustments to the storyboard (e.g., resizing 
pop-up windows, labeling the eontent and ehunking eaeh label into a self embedded hypertext window) 
aeeording to the feedbaek from the partieipants. Sueh adjustments eontinued until at least 4 SMEs agreed with 
the design approaeh. Based on the feedbaek reeeived from the seeond Delphi session, the remaining four 
funetional and instmetional features of the PSS support modules were finalized as shown in Figure 3. The 
operationalization of these features will be deseribed in the prototype stage. 




Interactive contents module Skill builder module Record keeper module On-line support module 



Stage 3: Prototype. In the prototype stage, the PowerPoint based storyboard was eonverted into a fully 
funetional prototype based on the proposed design strategies and data eolleeted in both assessment and design 
stages. The development tasks involved in this stage were, for the most part, teehnieally driven, and involved 
tasks sueh as database integration and system module assembly. Data in this stage were eolleeted from users 
using a series of one-on-one usability tests to (1) deteet any usability problems assoeiated with system 
navigation and interaetions, (2) determine whieh features in the system prototype eould be improved to beeome 
more workable (or eomplete) for the field testing, and (3) assess users’ initial reaetion toward the prototype. The 
usability test provided users with a set of navigational tasks, and the users were then observed to assess how 
they performed the tasks, what errors they made, and how they reaeted to the prototype. 

For eonvenienee and eost purposes, seven partieipants from the original partieipant pool of thirteen 
who resided in the Chieago area were seleeted for the usability test. Determination of this sample size was based 
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on Dumas and Redish’s (1999) usability guidelines, whieh reeommend five to twelve partieipants to attain 
reliable usability information to validate the prototype. The usability test followed the task-based usability 
proeedure, and was used to measure the time needed to eomplete a set of required tasks, reeord the number of 
steps needed to eomplete the tasks, as well as to reeord any errors produeed (Dumas and Redish, 1999). At the 
beginning of this test, we briefly introdueed the system to eaeh partieipant. Onee the partieipant understood the 
purpose of the prototype and its funetionality, he/she was asked to perform a set of tasks that required 
interaetion with the system's funetions. An evaluation aid eontaining evaluation tasks, operating questions, and 
a quiek referenee to the system interfaee was provided to guide the evaluation aetivities. After the usability test, 
a brief diseussion with the partieipant was earned out to aseertain overall feelings about the prototype and to 
solieit suggestions for improvement. 

Major findings of the system usability tests ineluded navigating and finding information throughout the 
matrix and its relevant eontent information from the knowledge database, interpretation and aeeuraey of visual 
information representation, and general use of navigation panel (e.g., seareh, print, bookmark, save, system 
module navigation). 

Stage 4: Planning. In the planning stage, the major task was to prepare the system for the field test. Therefore, 
data eolleetion in this stage was mainly a review of system dissemination proeedures. In reviewing the system 
dissemination proeedures, three major tasks were eonsidered: system requirements planning, system assembling, 
and system paekaging. Eaeh task involved a number of faetors that eoneemed the system delivery meehanism 
and playbaek. The proeess of system requirements planning involved review of the user profile survey data and 
phone ealls to the partieipants’ sehool eomputer lab administrators to inquire about their sehools’ teehnology 
infrastmetures. Based on the information gathered, dissemination eonsiderations were made to ensure the final 
system prototype would be eompatible with the partieipants’ eomputers. These eonsiderations inelude use of a 
eompatible medium to paekage the system prototype (i.e., CD-ROM in MAC and PC hybrid format); simplified 
installation proeess without the need for a eomputer administrator aeeount (to bypass the eomputer’ s register 
system); preseanning for viruses to ensure safe system operation; and transferability of user reeord files so that 
partieipants eould use the system both at work and at home without losing their development data. Based on 
these eonsiderations, the final system prototype was eompiled into a eompressed installer on a hybrid CD-ROM. 
A stand alone quiek start jobaid (in addition to the system’s embedded help system) to help the partieipants use 
the system prototype was also ineluded on the CD-ROM. 

Stage 5: Action. In the aetion stage, the main task was to implement the system prototype in the partieipants’ 
work environment to identify potential praetieal eonstraints that eould hinder the use of the system in a real- 
world setting. Prior to implementing this stage, a 30-minute training session on the use of the system was 
delivered to partieipants to help them beeome familiar with the system usage. 

Following the training session, all 13 partieipants were asked to use the system whenever they 
eneountered elassroom management problems with their students over the eourse of the subsequent two months. 
The proeess of asking partieipants to address real-life problems as a learning exereise is similar to aetion 
learning. As sueh, we divided the partieipants of this study into three groups based on their sehool affiliation, 
and the projeet team invited eaeh group to form an aetion learning set where group members eould eollaborate 
via the system prototype in developing intervention strategies for their students’ behavior problems. A 
behavioral psyehologist from the primary researeher’s eompany was designated as the set advisor to faeilitate 
the aetion learning proeess. While the partieipants were earrying out the aetion plans they had developed for 
their students’ behavior problems, they addressed the issues of implementation with the set advisor and their 
group members. 

In the beginning of the aetion stage, the set advisor ereated a “Seavenger Hunt” through some of the 
system modules to familiarize partieipants with its eontents. It was intended to be eompleted quiekly, yet also 
ensure that all partieipants viewed important seetions of the system. It was also intended as a tool to faeilitate 
thinking about how they eould use it in their groups. 

After the partieipants were familiar with the system, all three aetion groups were eneouraged to use the 
system as often as they wished. Weekly on-line diseussions were eondueted by the set advisor to answer 
partieipant questions that arose in the proeess of applying the interventions in their elassroom environments. 
Overall, aeeording to the set advisor, partieipants eommented that the matrix metaphor provided a quiek 
referenee for their routine elassroom behavior management tasks. Partieipants used the matrix to either support 
and eonfirm the intervention strategies they developed from their baekground knowledge or find alternative 
interventions for problems they eneountered in the elassroom. They reported feeling more eonfident and 
informed when using the system to make intervention deeisions and when preparing to seareh for information 
from both the Internet and book resourees. Partieipants reported (to the set advisor) that the elaborated and 
induetive way of finding relevant information for their elassroom behavior management issues (e.g., from 
behavior matrix to interaetive eontents to reeord keeper) gave them a sense of eontrol over the information. 
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Also, the structure of the information (i.e., each intervention included in the behavior matrix included a 
description of the intervention, when to apply it, how to implement it, and how to evaluate it) provided 
assurance and credibility of the information queried. 

Another noteworthy finding was that, according to the participants, the behavior matrix was the key 
feature that integrated the system modules. Participants would use the matrix to diagnose the issues in their 
classroom with other system modules serving as supporting resources to help them understand and manage their 
interventions. Furthermore, the format and information query structure provided in the behavior matrix guided 
them to systematize their development of interventions (e.g., degree of intrusiveness) and to create a dialogue 
for exchanging ideas with other teachers as well as with parents. Participants were able to discuss, refer, and use 
their interventions based on a common understanding that was laid out in the matrix. When applying the 
behavior matrix, the participants also used other system modules as supplements to help them understand the 
background information for the interventions and to retrieve more relevant information or strategies that could 
be used to develop more thorough interventions. For example, participants would start with the behavior matrix 
to help them recognize the problem in context and then systematically develop the intervention strategies 
obtained from the matrix. If more information were needed, participants would explore the strategies further 
through a direct link to the interactive contents and skill builder modules to learn additional information about 
the developed intervention strategies. Information gathered could also be saved in the record keeper module for 
future reference. 

However, concerns were also raised when applying the list of interventions and information presented 
in the system. In particular, participants were concerned about the rigidness of the matrix and lack of 
environmental components in the system. Participants felt that the matrix design, by providing quick cross 
reference and instant possible solutions to the target behavioral problems, at times it forced them into choosing 
interventions that were a reflection of the situation as it had occurred. That is, the one-to-one matching process 
(one behavior to one intervention at a time) inherent in the grid design approach could potentially lead a 
participant to feel the suggested intervention was dictatorial; this was quite evident because the situation was 
somewhat? ill-defined and could not be justified without contextual information. Additional grids such as 
environmental conditions and cognitive conditions of the child may be necessary to give the system a more 
comprehensive look. In addition, tutorials and instructions that advise new users about the adoption of the 
interventions should be provided to minimize the authoritative look of the system. 

Stage 6: Audit. After the field test, questionnaires on system impact, interviews of participants’ experiences and 
perceptions, and analyses on activity logs were carried out to assess the system design framework for supporting 
participants in making informed decisions, and to create a better way of integrating a fully developed system 
into the classroom environment. The questionnaire included 52 Likert response scale, short- answer, and open- 
ended items. These questions were derived from conversations, informal interviews, previous research and 
literature reviews (e.g., Redmond-Pyle, & Moore, 1995; Shneiderman & Plaisant, 2004). The purpose was to 
elicit participants’ opinions about the system and to characterize the overall value of the system in a quantitative 
manner. The questionnaire was composed of five constructs: (1) system operability and appropriateness for 
participants’ work environments, (2) system navigation and orientation, (3) system interface and content 
arrangement, (4) overall system usefulness, and (5) readiness for system adoption. Upon completion of the 
questionnaire, each participant was interviewed to gather more in-depth, qualitative information regarding his or 
her experience with the system. 

All 13 participants answered the survey at the end of the field test. Overall, participants had positive 
reactions to the system prototype and most of them indicated they would continue to use the system when it is 
fully developed. Specifically, they felt that the matrix metaphor was easy to use and not overly complex. They 
reported that the interaction and feedback provided adequate information to inform their actions. The grid based 
interface was seen as legible and promoted awareness of features that were embedded in each system function. 
The inadequate support and help to other system modules (i.e., skill builder and record keeper), however, 
prevented them from further exploring the system in depth. This finding was further confirmed with the system 
log. The participants’ system logs showed that the embedded help function was often accessed within the first 
five uses of the system with a mean staying time (before jumping back to system modules) of 42 seconds. 
Additionally, the logs? revealed participants’ expectation about each system module’s functionality. The 
participants were interested in identifying the quickest way that the system could support their work. Therefore, 
the system functions (e.g., behavior matrix, interactive contents, on-line support) that were capable of providing 
the fastest and easiest way to help users solve a problem became the most-used modules during the field testing. 

Upon completion of the survey questionnaire, follow up interviews were conducted to elicit more 
detailed information concerning system improvement. Each participant was interviewed via phone; these 
interviews lasted from 10 minutes to 15 minutes. Prior to the interview, participants’ answers to survey 
questions that addressed views on system adoption were reviewed by the participants and primary author to 
establish the question context quickly. Overall, the findings can be summarized into two major approaches that 
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concern the next iteration of system development: authority of eontent and user eustomization of system 
features. Some partieipants felt that the eontent presented in the system was written in "aeademie" language, and 
they suggested the eontent be revised so that they might more easily transfer what they had learned. In addition, 
the partieipants questioned the “fitness” of the intervention strategies listed in the matrix. Partieipants, 
partieularly those whose responses indieated non-adoption of the system in the future, were eoneemed with the 
rigidness of the system’s grid interfaee and hierarehieal strueture. They felt the matrix inferred with what eould 
be done when the issues are eomplex and require multiple aspeets. Without proper guidelines and training, they 
were eoneemed that the system eould easily beeome a “reeipe-like system,” partieularly for less experieneed 
users. Therefore, they suggested that additional instmetions or hints sueh as the ineorporation of environmental 
eoneem and individual differenees should be provided to help users make better deeisions, and the intervention 
list should be reorganized to provide a better fit for their elassroom environment. 

Sueh suggestions lead to the question of user eustomization. Most partieipants would like the matrix to 
be user eustomized, with allowanee for additional eontent grids and interventions. Also, an individual’s behavior 
matrix might be generated so that eould the user is able to eustomize the interventions to individual students and 
thereby integrate the system with eaeh student’s lEP. 

Conclusions 

The results of this study yielded the following eonelusions: 1) Overall, subjeets eonfirmed that the 
format of the matrix provided them with a quiek referenee and a guideline to help them more aeeurately 
eategorize their students’ dismptive behaviors and apply proper intervention strategies. The reasons given were 
the matrix’s grid and hierarehieal stmeture whieh enabled users to systematieally and sequentially seleet and 
organize information and strategies that best fit the problem eneountered. 2) Subjeets reported that by using the 
Behavior Matrix system as a guide to develop their intervention strategies, they were able to establish eommon 
ground and eommunieate with parents and regular elassroom teaehers using the same language. When there was 
a need to inerease the degree of intrusiveness of the interventions, all partieipants were able to look at the matrix 
and eome up with a deeision about whieh intervention strategy they would apply. While all subjeets had a very 
positive reaetion to using Behavior Matrix, there were eoneerns abut its praetieal implementation in the real-life 
setting: 3) The ehange proeess is very long term — while eoneeptual understanding and refleetion on the 
behavior management of others eomes fairly rapidly, the development of those intervention skills (the ability to 
refleet in aetion) requires long term effort. 4) Central to the ehange proeess is the ability to diseuss strategies 
and teehniques. Behavior Matrix may be useful for the users’ just-in-time problem solving, but its rigidity and 
grid format may inhibit their deeper understanding of the underlying prineiples of the applied strategies and 
teehniques. 5) Also eentral to the ehange proeess is ready aeeess to the Behavior Matrix (to models and 
refleetions on intervention strategy development) — ideally aeeessible from home or other non-sehool sites, e.g., 
libraries - whieh eould not be assured. 

In addition, the adoption of developmental researeh methodology has shed some light on the proeess of 
instruetional system development. Though the present researeh methods resembled the formative evaluation of 
an instruetional produet, information eolleeted during developmental researeh is primarily used to inform 
instruetional designers of how partieular problems have been solved or overeome (Reigeluth & Friek, 1999). 

The three design approaehes proposed in this study, eombined with their theoretieal assumptions, were devised 
and justified through a series of inquiry, analysis, development, and test proeesses. The shaping of an 
instruetional produet during its development proeess provides an opportunity to better understand how 
instruetional prineiples ean be used to inerease that produet’s leamability. In addition, this proeess uses 
eontinual revision to attain effeetiveness, feasibility, and aeeeptability for the produet. Thus, this shaping 
proeess provides one analytieal method for earrying out the developmental researeh methodology. 
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Abstract 

As a computer-based interactive guidance and information-support system, performance support system (PSS) has 
been used to support learning and performance in various settings. In this study, major interface design approaches 
for PSSs are reviewed and critical shortcomings of technology-driven PSSs are identified. In addition, a new design 
framework for PSS based on Ausubel’s advance organizers theoretical framework and a multidimensional 
information display structure are proposed. This new design framework. Matrix Aided Performance System, was 
field tested and the results reported in this paper. 



Introduction 

Performance support systems (PSS) have been adopted as an alternative to the design and development of 
computer-based learning for several reasons. A PSS has the ability to (1) offers advice and support tutorials for 
problem-recognition, concept identification and task management; (2) allow users to easily access carefully selected 
and organized information resources; and (3) provides analytical tools for users to evaluate their action plans. In 
other words, a PSS’ flexible and powerful modular design approach gives PSS designers a variety of options to 
create performance support tools most appropriate for the users’ work environments. A PSSs’ unique toolset design 
approach allows designers to incorporate multiple learning and performance modules (e.g., domain knowledge 
database, advisory system) to improve users’ job performance (Barker & Banerij, 1995; van Schaik, Pearson, & 
Barker, 2002). 

While the toolset design approach of PSSs offers a great deal of flexibility that many PSS designers desire, 
to integrate all system modules, designers must work hard to develop a unified system interface (Barker, & Banerji, 
1995; Gery et al. 2000; Laffery, 1995; McGraw, 1995; Raybould, 1995, 2000; Ring, 1994). A unified system 
interface, along with a coherent system architecture, are required to support effective user interaction with various 
system functions. The system interface works hand-in-hand with system functions that monitor and interpret user 
actions and provide appropriate responses to the user to ensure a positive user experience. A good example is 
Internet surfing using the web browser; the user does not need know HTML or how the Internet works in order to 
search and retrieve information. In this instance, the Internet browser serves as the unified interface that allows users 
to locate information they need from various sources in a variety of formats. 

Gery (1995) emphasizes the importance offering “intrinsic support” to users; she contends that in order for 
users to focus on their task at hand, the interface design must allow user to control their interactions with the system 
with ease. The goal of a unified system interface is to allow users to interact with, generate, store, and retrieve 
information contained in various system modules without sensing disparity or disconnection. 

Overall the years, there have been significant improvement in interface design, and various guidelines for 
addressing various interface design issues such as user and system interactions, navigation, orientation, and 
information storage have been offered. However, there are still two major issues concerning support for user 
performance that have yet to be addressed: 

1 . Comprehension-PSSs that employ a variety of advanced technologies may have powerful system 
functions and provide users a wealth of information in various modules, but they may result in a 
Steeper learning curve that requires the users to possess a higher level of content and system 
knowledge before they can comfortably use such a system. 
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2. Scaffolding- Another PSS design challenge is to provide guidance for the users and to help them 
anchor their knowledge and skill acquisitions in meaningful contexts. With a thoughtful interface 
design, some PSSs may be able to simplify user navigations and minimize the prerequisite knowledge 
for users to effectively use the system. However, there remains the problem of helping users to 
integrate queried information in the context of their work environment and critically examine the 
usefulness of that information. 

To address these design shortcomings, we are proposing an alternative design framework called Matrix 
Aided Performance System (MAPS). The design concept of MAPS is based on AusubeTs (1963) meaningful 
reception learning theoretical framework (i.e., advance organizers and subsumption theory) with the incorporation of 
PSS technologies and a multi-dimensional, matrix information presentation structure. The goal of MAPS is to 
operationalize the advance organizers concepts to create a truly instructionally-guided PSS interface to support 
users’ cognitive activities such as information comprehension and synthesis, to scaffold problem diagnosis and 
solution creation, and to promote self-directed learning. 

How Works 

As described in the above section, the MAPS design framework combines the advance organizers 
instructional design strategy, PSS technology, and graphical user interface design principles. As a PSS design 
framework, MAPS adopts an interdisciplinary approach. We believe that to effectively support certain types of user 
task (such as those requiring understanding, synthesis and application of complex concepts), we can borrow from 
instructional design principles. In other words, the ultimate goal of MAPS is to enhance user performance, and 
pedagogy can be used as a means to achieve that goal. In some cases — depending on the knowledge domain and 
user performance objectives — MAPS can be used to support knowledge and skill acquisition prior to task 
performance, as well as a reference and task execution tool during job performance. 

To illustrate the MAPS design approach, a schematic, generic template is used to demonstrate how MAPS 
works (Figure 1). As shown in Figure 1, the content and action grids are used to present users an overall structure of 
a knowledge domain or the “big picture” before delving into details in individual result grids. This is based on 
Ausubel’s (1963) advance organizer theory. In addition, the matrix structure of information is an effective way to 
condense lengthy information and to aid in logical problem solving. The multidimensional information display 
enhances user comprehension of complex concepts and their application in appropriate contexts. 

Figure 1. Matrix Structure oiMAPS 



Level B. Content grid (2™ level hierarchy) 
Level A. Content grid (1st level hierarchy) 



C. Action grid (problem 
solving strategies, 
predicting variables, 
simulation values) 



► D. Result grid 
(outcomes, applications) 



i Expansion grid 
V"^ (additional content/ 
action hierarchy) 



As shown in Figure 1, a generic MAPS interface contains four major grids and each grid can be expanded 
to accommodate the unique requirements of different knowledge domains. Grids A and B are content grids and they 
are used to display the hierarchical structure of the content materials. Much like a table of contents, grid A is used to 
present the first-level headings of the content or knowledge/skills, and grid B can be used to display the subheadings 
or further breakdowns of the knowledge/skills. Additional levels (i.e., 3^^ level subheadings) can be added based on 
content analysis and instructional needs. Each content grid can be color-coded and/or use a different font style to 
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provide visual cues to highlight the structure of the information presented. Information in grids A and B serves as 
the advance organizers for the learning materials to raise learners’ cognitive awareness of the relationships (whether 
it is hierarchical, causal, conceptual, or comparative) among content units. 

Across the top of the matrix is grid C. It presents action information, such as possible problem-solving 
strategies to support users’ problem-solving activities, predicting variables for analytical type performance support, 
simulation values for laboratory experiment or operations management decision support, and diagnosing steps for 
engineering troubleshooting. Each action item in the action grid corresponds to one or several rows of information 
in the content grids so that users can see the utility of the knowledge presented in the content grid. The arrangement 
of action information (displayed from left to right) is ordered by the range of effects or the degree of power of the 
actions/strategies, and the order can be determined by experts in the respective fields so that users can experience the 
experts’ problem-solving process. This visual presentation of content and action information serves as a cognitive 
framework and provides a mechanism for users to externalize their meta-cognitive processes and, hence, open them 
for evaluation. The matrix enables the users to compare their own problem-solving processes with those of an 
expert and other users through peer interaction functions in the PSS. 

Grid D presents the result of the action such as detailed descriptions and applications of problem-solving 
strategies and/or variations in strategy applications depending on the context. This grid will show whether an action 
is appropriate for a particular component information/skill. Due to the limitation of screen resolution and font sizes, 
each cell in the result grid can present only dyadic information (i.e., “yes” or “no”). However, this information 
display restriction can be overcome with hypertext technology; by allowing user interactions with the result grid 
(e.g., clickable cells), additional information can be retrieved and displayed for the user. 

Two working examples. Behavior Matrix and Customer Communications Matrix, are presented below to 
demonstrate applications of the MAPS design framework. The Behavior Matrix, as shown in Figure 2, is a behavior 
management tool for helping k-12 teachers diagnose and manage children with behavioral problems. This behavior 
management matrix was designed in collaboration with content experts to provide in-service teachers as well as 
parents with a condensed behavior management guide that compares a variety of intervention types and their 
applications to address specific behavioral problems. Teachers can use the Behavior Matrix to learn a systematic 
process for solving various behavioral problems. The Behavior Matrix can serve as a starting point to help in-service 
teachers and parents emulate the experts in systematically diagnosing and managing their students or children’s 
behavioral problems. Also, it can help establish a communication bridge among parents, educators, and experts by 
helping them use the same terminologies and “speak” the same language. 

Figure 2. MAPS example (Behavior Matrix) 
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In addition to the Behavior Matrix, another working example is shown in Figure 3. Adapted from 

Knapp’s (1999) customer communication styles. The Customer Communications Matrix displays a hierarchical 



152 






structure of caller communication types (non-aggressive, mixed, aggressive) and specific communication styles 
(passive, chatty, panicky, complaining, domineering, and explosive) in the content grids on the left-hand side of the 
matrix. Also in the content grid are several indicators for each communication style to help call center staff better 
gauge callers’ underlying emotions based on their verbal behaviors. Once the caller’s primary communication style 
has been determined, the call center staff can refer to the action and result grids for guidance on customer interaction 
strategies appropriate for the situation. 

Figure 3. MAPS example {Customer Communications Matrix) 
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Formative Evaluation of Behavior Matrix 

The Behavior Matrix was evaluated by a group of target audience (n=13) to determine users’ perceptions of 
the Matrix design approach in a realistic setting and the effectiveness of this tool in supporting their performance. 
Another objective of this formative evaluation was to identify any design issues that could hinder user experience 
and performance. 

Each subject was asked to use this system over the course of four weeks and as often as they desired. A 
user-perception survey and individual interviews were conducted after the four- week period to explore the subjects’ 
experiences using the system and the effectiveness of the system in enhancing their ability to manage their students’ 
behaviors. The user-perception survey included 49 Likert scale items, and 3 open-ended items. The overall 
Cronbach’s Alpha value of the survey is .84. Additionally, principal component analysis with varimax rotation 
method revealed 9 factors with eigenvalues greater than one (see Appendix 1). These results provided evidences of 
reliability and construct validity. 



Findings 

Findings on length of use and adoption intention and motivation 

Subjects were asked to indicate average time they spent on the system every week. The majority spend 
either 1-3 hours per week (five participants) or 3-5 hours (five participants). Only two participants spent more than 
5 hours, and one less than one hour. The participant who spent less than one hour noted the reason: her computer 
was too slow and it was inconvenient for her to use the system any time during the day. 
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The three open-ended questions explored participants’ reasons to continue or discontinue using the system 
including non-system aspects such as time, hardware, school policy that contributed to their decision to discontinue 
using the system. Overall, a majority of the participants who intended to continue using the system indicated the 
system’s easy access and quick referencing function contributed to their decision. Here are a few comments from the 
participants: 

“It was convenient” 

“I loved the matrix design. It was a fast and fairly easy way to reflect on my thinking.” 

“It is consistent, once you have figured out how it works.” 

“I liked being able to have all of the intervention recorded [the matrix’s save function] so that I could go 
back and reference them if I needed to. I liked the ”dof’ button. I enjoyed having an elaborated 
explanation within the cell [hyperlink to knowledge base] so I could see what kind of strategies I can use.” 
“It was extremely easy to use! !” 

While the participants liked the usefulness of the matrix’s design and the ease of use, they identified several 
areas of improvements: 

“At times the amount of information [information contained in the pop-up windows] was presented in an 
overwhelming fashion. I wished it could be arranged in a more organized fashion.” 

“The links [hyperlinks contained in pop-up windows] are not clearly identified. It is difficult to navigate the 
discussion page and to read text in the body of the messages.” 

“The set up of all the dots was a bit confusing. It was hard to tell which dot was going to which 
interventions/behavior. It would be very helpful if color is added to the dots or change their shape.” 

“More behaviors and interventions are needed!” 

Lastly, there were some concerns that were not due to the system. Some participants were reluctant to adopt the 
system due to circumstantial reasons such as lack of technical support to use advance system functions (especially 
the database functionality) and concern of school district’s policy on the use of the system in their classrooms: 

“I prefer talking to people face-to-face or at least in a way that I can hear a voice.” 

“It [the list of interventions] needs to be approved by our district before I can use it for my lEP [Individual 
Education Plan].” 

“Difficulty knowing which record was [related] to what student at times” 

“It was hard for me to use because I am not regularly logged onto a computer.” 

Findings on subject ’s perceptions of the system design 

The user survey and interviews revealed the following information. The subjects' perceptions of the 
system 's usefulness and appropriateness for their work environment, and the appearance of system interface and 
content arrangement: The subjects’ answers to the quantitative survey items and the qualitative interview questions 
on their overall experience with the system indicated that the Behavior Matrix'^ functionality, content quality, screen 
appearance, and interface design were well received. The majority of subjects felt that the screen layouts were 
effective in supporting their navigation in and interaction with the system. They felt that the system's interface 
(mouse, buttons, icons, and dialogue boxes) was intuitive and they felt secure and comfortable when interacting with 
the system. However, some subjects felt that the content presented in the Behavior Matrix was written in academic 
language, and the subjects suggested the content be revised so that they can more easily apply and transfer what they 
learned from the system to everyday classroom settings. In addition, the subjects had mixed reactions to the 
complexity of some of the system’s functions such as the database and text export functions. Most of them pointed 
out that more instructions on using these advanced system functions were necessary. 

The subjects’ perceptions of the ease of system navigation and orientation: The subjects agreed that it was 
relatively easy to get started and that they were able to begin using the system in a very short amount of time. As to 
the orientation of the system, the subjects felt that the grid layout was easy to follow and remember, and that the 
matrix design was effective in supporting their task performance. The straightforward information structure allowed 
them to executive information search and retrieval tasks in a reasonable number of steps. Furthermore, they pointed 
out that the knowledge base function was helpful and self explanatory. However, the subjects expressed some 
frustrations when they tried to explore more advanced system functions (record keeper and on-line support) through 
trial and error. 

The subjects’ perceptions of the system ’s operability: Overall, the subjects were satisfied with the system’s 
response time, its tolerance of user errors and its auto-correction functions. In addition, the subjects agreed that the 



154 




system accommodated the needs of both experienced and inexperienced users. However, the subjects had mixed 
reactions about whether the system should provide explanations and warnings about users’ errors. Some subjects 
felt that the system was fine without providing any feedback after users had executed a task while the others 
considered feedbacks were necessary. For example, when a user selects the Save function (see Figure 3), the system 
should confirm that the selected content has been exported. 

The amount of user training and support in the system adoption process: The majority of the users felt that 
the amount of user training and support was adequate. However, they pointed out that more time should be spent on 
the more advanced system functions. When asked about their willingness to continue to use the system after it was 
fully developed, eight subjects strongly agreed that they would adopt it. Other subjects were reluctant to adopt the 
system due to circumstantial reasons such as the lack of technical support to use advance system functions 
(especially the database functionality) and concern of school district’s policy on the use of the system in their 
classrooms. 

Subjects were also asked to record their activities using a log sheet and to install the usage tracking 
function. However, many subjects did not comply with these two data collection requirements because they did not 
have the time to manually enter information onto the log sheet and others forgot to install the tracking function. 

Only two sets of log sheets and interaction histories were collected after the field test. Information gathered from 
these two sets of data was excluded from data analysis. 



Discussion 

The study findings suggested that the Behavior Matrix was a valuable resource for the subjects. The 
subjects credited the system for helping them deal with behavioral management problems they encountered daily in 
the classroom and clearly conceptualize students’ behavioral problems so they can more systematically document, 
track and discuss students’ behavioral issues with parents and school resources teachers. Besides the useful content, 
the subjects found the system to be well designed. The matrix information structure and usable screen layout made 
the task of information search and retrieval intuitive and easy to remember. 

While the subjects’ feedback was positive and their motivation for adopting the system was extremely high, 
the findings revealed several areas of improvement. Provision of guided feedbacks so users can fully exploit the 
more advanced system features (e.g., provide user prompts or wizard function) and expansion of the system’s 
configurable commands and automated controls (e.g., the save command needs to be more intelligent when 
appending selected contents to the record keeper) to increase the flexibility and adaptability of the user interface are 
two improvements for enhancing the effectiveness of the system. Furthermore, offering an expert tutorial to 
improve users’ understanding of the domain knowledge and rephrasing content material using non-academic terms 
are other areas of improvement. 



Conclusion 

Interface design and human-computer interaction (HCI) have been areas of emphasis in computer- 
supported instructional systems and performance support systems where the graphical user interface design and 
system usability play major roles in the design process. Key concerns when developing a successful and acceptable 
user interface include: (1) the quality of icons, buttons, and layout to help users reduce information overload; (2) the 
consistency of interface presentation to lower the chances of confusion, discomfort, and fatigue; and (3) the adoption 
of an integrated, iterative design approach to continually measure the usability of the system’s human-computer 
interface. Such efforts in an HCI-oriented design approach would certainly ensure a usable interface for a software 
application, but they are inadequate for PSSs and other types of learning and performance-based instructional 
systems where learning and performance mastery is the goal. The requirements for a usable interface of a PSS 
should be extended to include scaffolding and support for user comprehension by presenting meaningfully 
integrated PSS modules. The MAPS design framework and working examples presented in this report are examples 
of pedagogy-driven PSSs with unified interface designs that are functionally and instructionally integrated. By 
carefully assessing and selecting content materials and system functions, and then integrating them using a matrix 
structure that serves as an advance organizer, the MAPS design framework provides both a usable interface and 
effective performance supports (i.e., comprehension and scaffolding). 

The MAPS design framework and working examples are examples of pedagogy-driven PSSs with unified 
interface designs that are functionally and instructionally integrated. Several design considerations and benefits are 
listed below to facilitate the adoption of the MAPS design framework. 
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1 . provide users a holistic view of the knowledge domains related to the problem and their inter-relationships (i.e., 
with advance organizers), 

2. help users recognize common problem patterns and problem-solving strategies (i.e., the hierarchical content 
structure and action grid), and 

3. provide usable interface to help users determine which additional system functions they can use to support their 
problem solving. 
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Appendix; Factor Loadings for Items Regarding Operability, Navigation, Interface, Usefulness, and Adoption 



Component Matrix(a) 




Component 


1 


2 


3 


4 


5 


6 


7 


8 


9 


System Operability: Compared to your usual approach to behavior manag 


ement, do you consider BM to be: 


1 . On the average, how much time do you spend per week on 
this system? 


0.88 


















2. More time-efficient for your development of behavior 
interventions 


0.75 


















3. More convenient for you to use 


0.53 


















4. More important for your group discussion 


0.62 


















5. More effective for you to use 










0.49 










6. Easier for you to manage your developed behavior 
interventions 










0.70 










7. Easier for you to interact with your colleagues 


0.51 


















8. Providing better communication between you and your 
students’ parents 


0.70 


















9. Providing less quality of discussion to your colleagues 


0.64 


















10. Not compatible with my intervention development 
approach 


0.81 


















System Navigation & Orientation: 


1 . The terminology was clear and precise. 












0.51 








2. 1 always know where to access the information 1 wanted. 


0.42 


















3. 1 feel secure and comfortable in interacting with the 
system. 


0.91 


















4. 1 feel the system responded in a consistent and predictable 
way. 




















5. 1 feel BM provides inadequate guidelines to help solving 
problem 












0.45 








6. 1 feel BM provides inadequate functions to facilitate 
discussions 


0.60 


















Interface & Content Arrangement: When using BM, you feel: 


1 . Screen and matrix layouts helpful 




















2. Amount of information that can be displayed on screen is 
adequate 








0.29 












3. Arrangement of information on screen is logical 








0.51 












4. Sequence of screens is clear 








0.57 












5. Screen design is appealing 








0.22 












6. Amount of graphics is appropriate and purposeful 








0.59 












7. Content headings are clearly labeled 








0.46 












8. Icons and dialogue boxes are not appropriately sized 




















9. More hints should be provided 












0.24 
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Overall System Usefulness 


1. BM helps me understand the nature of the behavioral 
problem 






0.53 














2. BM helps me recognize the obstacles associated with the 
behavioral problem 






0.25 














3. BM helps me generate hypotheses about the behavioral 
problem 














0.49 






4. BM helps me reason the hypotheses that I generated 














0.45 






5. BM helps me identify the facts of the behavioral problem 




















6. BM helps me identify the learning issues involved in 
behavior management 
















0.20 




7. BM makes me feel I can’t solve the problem based on the 
information given 


















0.41 


8. BM helps me have a clear understanding how I arrive my 
final interventions 
















0.36 




9. BM provides me better way to analyze behavioral problem 




0.54 
















10. BM makes behavior management easier to practice 
















0.23 




1 1 . BM prompts me to study the behavioral problem carefully 














0.24 






12. BM encourage me to identify critical features of the 
behavioral problem 




0.76 
















13. BM prompts me to give many considerations toward the 
behavioral problem 




0.34 
















14. BM helps me apply my understanding of the behavior 
problem to the interventions 




0.58 
















15. BM offers me better management of my thinking process 
toward the behavioral problem 
















0.30 




16. BM offers me better record keeping of my development of 
intervention strategies 


















0.30 


17. BM are mostly for people who have good understanding 
of computer 






0.34 














System Adoption 


1. BMwill improve the outcome of classroom behavior 
management 










0.39 










2. BM is a tool suitable for classroom behavior management 


















0.29 


3. 1 would encourage others to use BM 




















4. 1 want more training on BM 










0.59 










5. 1 would like to continue to use BM 










0.46 










6. BM\n\W weaken teacher-student relationship 




















7. Overall, if fully developed, 1 would like to continue to use 
the system 




















Eigenvalues 


7.37 


2.22 


1.12 


2.64 


2.63 


1.19 


1.17 


1.09 


1.01 



BM = Behavior Matrix 
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Abstract: The ability to thinking reflectively and doing deep learning has been advocated as an essential 
educational goal in schools and colleges. However, an observation and a pedagogical challenge often found in 
classes is that students think at the quasi-reflective stage. This paper presents preliminary findings from a research 
which experiments with the use of weblog to facilitate students’ reflective learning in an introductory political 
science course. Utilizing Map of Learning, a reflective learning model, this study explores whether a web-based 
blogging environment will reinforce reflective thinking and whether students’ reflective thinking levels will predict 
their learning approaches. 



Introduction and Theoretical Background 



Reflection and Learning 

Reflection, as defined by Knapp (1993), is a means for reliving and recapturing experience in order to make 
sense of it, to learn from it, and to develop new understandings and appreciations. A review of the literature on 
reflection indicated its value in improving learning and professional practice at different situations, such as 
experiential learning, reflective practice in teacher education, professional education in counseling, and personal 
development (King & Kitchener, 1994; Kolb, 1984; Mezirow, 1991; Schon, 1983). 

Based on previous works on reflection. Moon (2004) has proposed a hypothetical model, named Map of 
Learning, to synthesize the role of reflection in human learning process. In this model. Moon identifies learning as a 
continuum ranging from the stage of “noticing,” “making sense,” “making meaning,” “working with meaning,” to 
“Transformative learning” (2004, p. 139). The previous two stages are surface learning where the learner simply 
memorizes new ideas, while beginning at the third stage the learner does deep learning by actively integrating new 
ideas into cognitive structure. In agreement with Mezirow (1991), Moon believes that it is reflection that upgrades 
learning from surface stages to deep ones. For instance, reflection is suggested to be the means of integrating the 
learning into the cognitive structure and relating it to previous knowledge; reflection on learning process enables the 
learner to take a critical overview and accumulate further understandings of the self or the knowledge, hence push 
him/her into the transformative learning stage. 

Moon’s model pinpoints the place of reflection in learning, but it hasn’t explicitly accounted for different 
levels or quality categories of reflection and whether levels of reflection will predict the learner’s contemporary 
learning stages (surface or deep). Specifically, since scholars (i.e.. Bond, Keogh, & Walker, 1985; Hatton & Smith, 
1995; Mezirow, 1991; Taggart & Wilson, 1998) have agreed on a pyramid-shaped model of reflective thinking 
where reflective process comprises multiple levels (e.g., association, integration, validation, and appropriation), an 
imperative question naturally posed is, “Will reflective thinking levels predict the learner’s learning approaches or 
stages?” A research in response to such a question is still missing in the field. In addition. Moon’s Map of Learning 
is mainly a hypothetical model that is in need of support and clarification from empirical work and in-field 
observation of reflective activities in a concrete learning environment. 

Blogging for Learning Reflection 

Journal writing as a means to reflection has been well discussed in the literature and becomes common in 
learning of different professions. More recently, weblog (or blog), as a new format of journaling tool, has captured 
the attention of education community. A weblog is a frequently updated website consisting of dated entries 
arranged in reverse chronological order (Walker, 2003). Quite a few researchers (e.g., Efimova & Fiedler, 2004; 
Stiler & Philleo, 2003; Williams & Jacobs, 2004) have started exploring the learning-related effects of weblog. For 
example, Stiler and Philleo (2003) incorporated reflective blogging into the syllabi of two undergraduate preservice 
courses and found that students using blog were more analytic and evaluative than those of the prior semester. 

These works offer some valuable discoveries on using weblog to support learning. However, more in-field studies 
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that focus on weblog as a reflective practice tool and the purposeful design of weblog-based journaling environment 
are still needed. 

Research Purpose 

This study intends to explore the reflective learning process in a weblog-based journaling environment. 
Specifically, the researchers focus on two research questions: (1) Will a web-based blogging environment reinforce 
participants’ reflective thinking? (2) Will participants’ reflective thinking level predict their learning approaches or 
stages, hence their learning achievements? 



Research Methods 

The current research is a longitudinal study that examines students’ usage of weblog over a regular school 
semester. Data was collected from a content analysis of weekly weblog journals, a self-report survey on students’ 
learning approach, and students’ course grades. 

Participants 

Fifty college students enrolled in an introductory political science course at a major American university 
participated in the study. All of them were first year students and one-third of them are female. Absence in self- 
report survey and failure to complete weekly weblog journals led to some subject attrition. The pre-study 
demographic survey indicated none of participants had used weblog before. 

Procedures 

Blogging was incorporated into the course syllabi. Specifically, Blogger (http://www.blogger.com), the 
most widely used weblog, was used in this study. A journaling structure guideline was given to students at the 
beginning of the semester. This structure guideline was developed by the researchers based on relative reflection 
theories - Dewey’s (1933) five phases of thinking and Johns’ (1992) model for structured reflection. Participants 
were told that the course requirements included the completion of reflective journals using Blogger and they must 
write at least one journal every week. All participants registered the Blogger accounts using pseudonyms assigned 
by the researchers. All participants attended a one-hour orientation to learn the usage of Blogger. 

Weblog journals were archived at the first and the last school week of the semester. Two sample journals 
in total were gathered (one at the second week and the other at the last week) and graded respectively for each 
participant. At the end of semester, participants did the Study Process Questionnaire (SPQ) (Biggs, Kember, & 
Leung 2001) that measured their learning approaches - surface versus deep learning. The researchers employed the 
Coding Scheme of Reflective Process by Wong, Kember, Chung, and Yan (1995) in coding weblogging journals 
and determining the participants’ reflective thinking levels. Two raters coded the journals. After reaching 100 
percent agreement on scoring the first 5 blog samples, all raters double-blindly scored the rest sampled journals. 

The inter-rater reliability is .83. 

Instruments 

Several potential reflective thinking coding schemes were explored to see whether they provided a suitable 
framework for reflection level evaluation. The coding scheme by Wong et al. (1995) was eventually used due to the 
following reasons: (a) the coding scheme was developed using the conceptual framework of Boud et al. (1985) and 
Mezirow (1991), thus matching the current research’s theoretical assumption on a pyramid-shaped model of 
reflective thinking; (b) the coding scheme was in-field tested and the reliability was .88; (c) the coding scheme 
offered a detailed rubric with clarification of evaluation criteria, hence making its application consistent across 
different raters and subject contents. The coding scheme classified reflective thinking into seven levels. From low 
to high, they are Attending to Feelings, Association, Integration, Validation, Appropriation, and Outcomes of 
Reflection. In the present study, these six levels were coded as rank-scaled score, ranging from 1 to 6, with 0 
referring to non-reflection. 

The Study Process Questionnaire (Biggs, Kember, & Leung 2001) is a 20-item Likert-scaled survey used to 
determine participants’ learning approach in two dimensions - Deep Approach (DA) and Surface Approach (SA). 
Each dimension is measured by 10 items. The dimension where a participant scores higher (either DA or SA) is 
considered as the dominant learning approach of that participant. The reliabilities for these two latent 
dimensions/factors were reported as 0.73 and 0.64 respectively. 
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Data Analysis 

A paired-sample t-test was conducted to investigate whether participants would demonstrate an 
improvement in reflective thinking level during the blogging process. Then a correlation analysis was used to 
examine whether participants’ reflective thinking level in journaling would predict their learning approaches (as 
indicated by SPQ survey result) and their learning performances (as indicated by their course grades). 

Findings 

A paired-sample t-test was conducted to compare participants’ reflective thinking level demonstrated in the 
first online journal and their reflective thinking level in the last online journal. The result is significant, t (27) = - 
4.44, p < .001. There is significant difference between the participants’ initial reflective thinking level (Mi = 2) and 
their reflective thinking level after weblogging (M 2 = 3). Additionally, it was found that in the initial online 
journaling, only six (20%) out of a total of 30 students can be counted as “critical reflectors” - demonstrating 
reflection at one or more of the later three levels, i.e. validation, appropriation, and outcomes of reflection (Mezirow, 
1991; Wong, et al. 1995). Then, it turned out that after weblogging 52% of this group of students (14 out of 27) 
demonstrated one or more of the later three reflective thinking levels in online journaling. Therefore, there is strong 
evidence suggesting that weblogging improved participants’ reflective thinking level. However, it should be noted 
that the current study didn’t involve a control group for experimental comparison. Therefore replicated studies are 
needed to confirm the positive effects of weblogging on reflection thinking development. 

Subsequently, correlation analysis was performed to examine the relationship between participants’ 
reflective thinking level (as indicated in online journals after weblogging), their learning approaches (DA or SA as 
indicated by SPQ survey result), and their course grades (continuous-scaled). The results indicated a significant 
positive association between participants’ reflective thinking level and their course grades: Spearman’s p = .40, p 
= .05. Therefore, the higher one’s reflective thinking level is, the higher his/her course grade is. It was also found 
that participants’ learning approach would predict their course grades, Spearman’s p = .54, p = .02. Specifically, 
participants with Deeper Approach in learning tend to get higher course grade. However, there was no enough 
statistical evidence suggesting that reflective thinking level may predict learning approach, Spearman’s p = .16, p 
= .61. Therefore, the results of the current study did confirm the claim of previous scholars (e.g.. King & Kitchener, 
1994; Kolb, 1984; Mezirow, 1991; Moon, 2004; Schon, 1983) on the value of reflective thinking in promoting 
learning performance. On the other hand, the study didn’t indicate enough statistical evidence to support Moon’s 
(1999) belief that it is reflection that upgrades one’s learning from surface stages to deep ones. 
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Abstract 

In this research we investigate a strategy that can support the comprehension of dynamic systems. 175 
students from seven sections of an undergraduate biology class at a large public university in US were asked to 
interact with instruction about the human heart. The instruction included text, static diagrams and animation 
developed using Macromedia Flash. Three conditions were set up - Control, Overview and Prediction. In the 
Prediction condition students were first provided an overview - using static diagrams and text they were given 
information about three temporally separate stages in the human heart cycle. Based on this information, students 
were asked to make some predictions about the behavior of the heart. Finally they interacted with the instruction and 
responded to the test questions. The Control group was not provided with any overview; they were taken directly to 
the instruction and asked to respond to the test questions. The Overview group was given the same overview as the 
prediction group but they were not asked to make predictions. Although no significant differences were found across 
conditions on the total scores [F (2, 94) = 2.606; p = 0.08], the trend for the scores on all four tests was in the 
direction expected - students in the prediction group performed better than the other two groups. These results are 
consistent with earlier research (Hegarty, Kriz, & Cate, 2003) 



Introduction 

Creating internal representations from reading expository texts is a complex and effortful process (Adams, 
1994; Kintsch, 1993; Sadoski & Paivio, 2004). Consider, for example, text about the human heart. First of all the 
learner has to be familiar with the relevant terminology - that is the learner should be able to identify various parts 
of the human heart. Next, the learner must know the functioning and purpose of individual parts of the human heart. 
For example, the learner should know about the functioning of heart valves, i.e. how the opening and closing of 
heart valves controls the flow of blood in the heart chambers. 

Learning becomes especially challenging when the expository text involves dynamic systems. Perhaps the 
most difficult part in learning about dynamic systems from reading expository texts is the understanding of the 
dynamic aspects of these systems. Different parts (sub-systems) in a dynamic system are causally linked to each 
other. These sub-systems work in synch to produce the overall effect of the system. Comprehension of the dynamic 
aspects of the functioning of such systems involves the ability to infer the state of one component of the system 
based on the states of other components of the systems. Such inferences can only be made if the learner knows how 
the different parts of the systems are coupled with each other. 

Previous research shows that comprehension of dynamic systems is a cognitively complex task. For 
example, in their research on learning about pulley systems (Hegarty & Just, 1993) proposed that learning about 
dynamic systems is a two step process. In the first step, learners integrate information locally - they read text, and 
create mini mental models about a component or a small group of components of the larger system. Global 
integration occurs only in the second step when learners integrate the mini mental models created previously into the 
mental model for the whole system. 
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Theoretical Background 



Effect of Text, Static Diagrams and Animations 

(Hegarty & Just, 1993) have looked at the effect of text and static diagrams on the comprehension of 
dynamic systems. While text or diagrams alone were found to be sufficient for learning more factual types of 
information about such systems, the authors found that a combination of both text and diagrams is particularly 
effective when it comes to understanding the dynamic aspects of the system. 

Animation has also been used in instruction to facilitate learning about dynamic systems, for example 
(Hegarty et ah, 2003). Research findings with regard to the impact of animation on learning are mixed though 
(Richer, 1993). In order to develop the instruction used in the present study, a pilot study (n=138) (Ausman et al., 
2004) was conducted to identify parts of the instruction where students performed below average. Animation was 
inserted at these locations in order to facilitate learning of the more difficult content. Instruction used in the present 
study employs text, static diagrams as well as animation to communicate the content. 

Process of Mental Animation 

Hegarty (1992) accounts for the mental processes involved in the comprehension of dynamic systems. In 
this research, the author provided a detailed account of the type of knowledge and learner processes that allow 
learners to infer functioning of dynamic systems that involve the interplay of several parts. The author proposed and 
tested two possible explanations: 

1 . Parallel Processing Explanation: Since in a dynamic system many components of the system work 
together (for example pulmonary and the aortic valves in the human heart open and close together, or the left and the 
right ventricles expand and contract together), a mental process that is isomorphic to the functioning of the system 
would involve mentally animating different components of the system at once. 

2. Piecemeal or Serial Processing Explanation: Learners try to understand the functioning of dynamic 
systems by animating the different components of the system “piecemeal”. In other words, this kind of processing 
involves decomposing the behavior of a complex system into smaller sub processes and then animating these sub 
process in sequence. To be able to explain the sub processes learners must understand the cause and effect relations 
between the various components of the system. This also means that to be able to explain the functioning of a 
component later in the causal chain, the learner must mentally run (or animate) the components and sub-process that 
occur earlier in the causal chain. Hegarty (1992) refers to this process as mental animation. 

Hegarty (1992) found evidence to conclude that learners (at least non-experts) do not process all the 
movements of the various parts of a system all at once (i.e. learners do not parallel process). Instead learners process 
the information piecemeal (i.e. serially) by inferring the causal chain of events in the sub-systems. 

The relationship between mental animation ability and spatial ability has been studied as well. For example 
(Hegarty & Kozhevnikov, 1999; Hegarty & Steinhoff, 1997; Mayer & Sims, 1994) have all found mental animation 
ability to be highly related with spatial ability of learners. (Hegarty & Kozhevnikov, 1999) have called this ability 
visual imagery - “the ability to form mental representations of the appearance of objects and to manipulate these 
representations in mind” (pg. 684). In the present study, learners' spatial ability is used as a covariate. 



Effect of Responding to Prediction Questions 

By means of three separate experiments, (Hegarty et al., 2003) compared the effect of the following 
conditions on learners understanding of dynamic systems: static versus animated diagrams; static and animated 
diagrams accompanied with verbal explanations; and predicting motion from static diagrams. On the basis of this 
research the authors postulated that asking learners to predict the behavior of a dynamic system engages them in the 
process of mental animation. This in turn helps their comprehension of the system in the following ways: (1) It 
forces learners to begin to construct mental models, albeit rudimentary. Learners work with these mental models and 
refine them as they go through the instruction. (2) It provides learners with information about what they understand 
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and what they do not understand about the system. (3) It induces learners to activate their prior knowledge of the 
system thus preparing them for the instruction that follows. 



In other works, responding to prediction questions forces learners to generate explanations about the 
functioning of the dynamic system. For example, (Chi, Leeuw, Chiu, & Lavancher, 1994) note that the act of 
generating “self-explanations” facilitates learning by inducing learners to integrate new knowledge into existing 
knowledge. (Chi et al., 1994) showed that self-explanation can not only facilitate acquisition of problem solving 
skills, but also declarative knowledge. 

One of the experiments in the (Hegarty et al., 2003) study investigated the impact of asking students 
prediction questions after they were provided with an overview (text and static diagrams). There was no condition in 
this experiment where students were provided with an overview (static diagrams and text), but were not asked to 
predict. In this study we have included an Overview condition. This allows us to separate the effects of extra 
information provided in the overview from the effects of mental animation that is induced when a learner responds 
to prediction questions. 



Method 



Participants 

About 175 students from seven sections of an undergraduate biology class at a large public university in US 
were asked to interact with instruction about the human heart. The average age of the participants was about 20 
years; majority of them being sophomore and junior year students. There were more female participants than male 
participants. The participants were informed that the material covered in the instruction would be on the final exam. 
On an average the students performed well on the tests that followed the instruction. 



Instruments 

In the first part of the study participants were asked to take the spatial ability test (Ekstrom, French, 
Harman, & Derman, 1976). This tests consists of paper folding tasks and card rotation tasks (Ekstrom et al., 1976). 
Students also took the test for prior knowledge in human anatomy (Dwyer, 1977). Questions on the prior knowledge 
test, tested participants’ general knowledge of the human anatomy. 

In the second part of the study, students in all three groups took the following tests (Dwyer & Lamberski, 1977) 
after completing the instruction. 

1. Drawing Test: The objective of the drawing test was to evaluate participants’ ability to construct 
and/or reproduce items in the appropriate context. The drawing test provided the students with a 
numbered list of terms corresponding to the parts of the heart discussed in the instructional 
presentation. The students were required to draw a representative diagram of the heart and place the 
numbers of the listed parts in their respective positions. For this test the emphasis was on the correct 
positioning of the verbal symbols with respect to one another and in respect to their concrete 
referents. 

2. Identification Test: The objective of the identification test was to evaluate participants’ ability to 
identify parts or positions of an object. This multiple-choice test required students to identify the 
numbered parts on a detailed drawing of a heart. Each part of the heart, which had been discussed in 
the presentation, was numbered on a drawing. The objective of this test was to measure the ability of 
the student to use visual cues to discriminate one structure of the heart from another and to associate 
specific parts of the heart with their proper names. 

3. Terminology Test: This test consisted of items designed to measure knowledge of specific facts, 
terms, and definitions. The objectives measured by this type of test are appropriate to all content 
areas which have an understanding of the basic elements as a prerequisite to the learning of concepts, 
rules, and principles. 
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4. Comprehension Test: Given the location of certain parts of the heart at a particular moment of its 
functioning, the student was asked to determine the position of other specified parts or positions of 
other specified parts of the heart at the same time. This test required that the students have a thorough 
understanding of the heart, its parts, its internal functioning, and the simultaneous processes occurring 
during the systolic and diastolic phases. The comprehension test was designed to measure a type of 
understanding in which the individual can use the information being received to explain some other 
phenomenon. 

5. Total Test Score: The items contained in the individual criterion tests were combined into a 
composite test score. The purpose was to measure total achievement of the objectives presented in 
the instructional unit. 



Instruction 

At the beginning of the instruction phase of the study, all three groups were provided a labeled 
diagram of the human heart (Figure 1). 



Aorta 



. Pj]monary Artery 
-A - - Pu tin ona ry Veins 



Left Atrium 




Right Ventricle 



De- oxygenated Blood 
Oxygenated Blood 
Heart Valves 



Figure 1. Labeled diagram of the human heart. 

Prediction and Overview groups were provided with the following information about three stages of the human heart 
that are temporally separate: 




Blood flows into the heart from two sources. Blood 
returning from the body flows into the right atrium 
through the superior and inferior vena cava. Blood 
returning from the lungs flows into the left atrium 
through the pulmonary veins. While the heart is still 
relaxed, the blood entering the right atrium flows 
past the tricuspid valve and into the right ventricle. 
On the left side of the heart, blood entering the left 
atrium flows, flows past the mitral valve, and into 
the left ventricle. The ventricles fill up first, and 
then the atria fill up. 
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Blood 



Blood to Lungs 




bntract 



Ventricles Contract 



When the heart begins to contract, the atria contract 
together and then the ventricles contract together. 

As the atria contract, the myocardium (heart muscle) 
of the atria squeeze down against the blood, 
increasing the pressure inside the atria, and pushing 
the blood into the ventricles. The ventricles are now 
swollen with the blood they held initially, plus the 
blood added by the contraction of the atria. Almost 
immediately after the atria stop contracting, the 
ventricles will start contracting. As the myocardium 
of the ventricles contract, the pressure inside the 
ventricles increases, and some blood will try to flow 
back into the atria. This is prevented by the closure 
of the tricuspid and mitral valves. As the pressure 
inside the ventricles continues to increase, the 
pulmonary and aortic valves (semilunar) valves are 
forced open. Blood in the right ventricle is forced 
through the pulmonary valve, into the pulmonary 
artery, and toward the lungs. Blood in the left 
ventricle is forced through the aortic semilunar 
valve, into the aorta, and toward the rest of the 
body. 

Blood in the pulmonary artery enters the lungs. 

From the lungs it goes into the pulmonary veins, and 
back to the left atrium. The blood in the aorta flows 
to the rest of the body. From the body into the 
superior and inferior vena cavas, and back into the 
right atrium. Thus the cardiac cycle repeats. 




On reaching maximum contraction the ventricles 
begin to relax. Pressure inside the ventricles starts to 
decrease and some blood is sucked back from the 
pulmonary artery and aorta into the right and left 
ventricles respectively. This is prevented by the 
closure of the pulmonary and aortic valves. 



to Relax 



Ventricles begin to Relax 



Figure 2. Overview of three stages of the human heart. 

In addition to the images and text shown above, the Prediction group was asked to respond to the following 
prediction questions: 

1 . Imagine you are in a tiny submarine that has just entered the heart through the right atrium. Trace the path 
you would travel through the heart and body if you travel with the rest of the blood, finally returning to the 
right atrium. Students were asked to drag and drop parts of the heart and put them in the right sequence. 
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2. When you go to the doctor's office, the nurse measures two blood pressures - one higher pressure and one 
lower pressure (for example, 120/80). 

a) What are the ventricles doing when your blood pressure in 120? 

b) What are the ventricles doing when the blood pressure is 80? 

3. A person walks into your clinic with swollen arms and legs. Identify which of the heart valves of this 
person is not functioning properly. 

4. As the ventricles contract, what do you think happens to: 

a) The tricuspid and bicuspid valves. 

b) Aortic and semilunar valves. 

5. At the point in time when the auricle pressure becomes greater than the ventricle pressure, what is the 
position of the tricuspid and mitral valves? 

Students in all three groups interacted with the instruction about the human heart (Dwyer & Lamberski, 1977). The 
instruction employs a combination of static images and animation to support the instruction. 



Procedure 

The study was conducted in two parts. Students in all three groups were emailed a Web link for Part 1 of 
the study. In this part of the study, participants completed a demographic survey, and took two tests: a spatial ability 
test and a prior knowledge test. Participants could complete this part of the study on any computer, anywhere, and at 
any time before they came to the lab for the second part of the study. Students typically completed Part 1 in 20-30 
minutes. 



In Part 2, students were randomly assigned to three conditions - Control, Overview and Prediction. This 
part of the study was carried out in university labs in the presence of the researcher. Upon arrival students were 
provided instructions by the researcher. During this part, participants first interacted with online instruction about 
the human heart (30 minutes-45 minutes). After this they took the four post-tests (about 5 minutes each, total 15-30 
minutes). 

At the beginning of the instruction phase of the study, participants in all three groups were shown a labeled diagram 
of the human heart (Figure 1). They were given the following instructions: 

'"Before beginning the tutorial, carefully study the labeled diagram of the human heart given 
below. Try to learn the names of as many parts as possible. We will be referring to the names of 
these parts all through the instruction."" 

Participants were given 2 minutes to study the labeled diagram. 

1. Control group: After spending two minutes studying the labeled diagram of the human heart (Figure 1), students 
in this group were directed to the online instruction. After this they took the four post-tests. 

2. Overview group: First, the students were shown the labeled diagram of the human heart (Figure 1). After this they 
were shown static images of three phases of the human heart along with text describing the images (Figure 2). 
Participants were given the following instructions: 

"It is important that you study this information very carefully, this may help you learn better 
from the human heart tutorial that follows. Try to understand the functioning of the human 
heart in such a way that you can explain it to someone else."" 

Finally they interacted with the instruction and responded to the four post-tests. 

3. Predication group: First, the students were shown the labeled diagram of the human heart (Figure 1). After this 
they were shown static images of three phases of the human heart along with text describing the images (Figure 2). 
Participants were given the following instructions: 
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‘'It is important that you study this information very carefully: this may help you learn better 
from the human heart tutorial that follows. Try to understand the functioning of the human heart 
in such a way that you can explain it to someone else. On the next page you will be asked to 
make some predictions about the functioning of the human heart.'' 

This was followed with prediction questions. Students could access the labeled diagram of the human heart 
as well as the static images of three phases of the human heart along with text describing the images when they 
answered the prediction questions. In the end students interacted with the instruction and responded to the post-tests. 

In all 104 students participated in Part 1 of the study. 137 students participated in Part 2. For the final 
analysis, we only used that subset of participants who participated in both parts of the study. This number was 94. 



Scoring 



While the scoring procedure for spatial ability, prior knowledge, identification, terminology, and 
comprehension tests is objective, the scoring procedure for the drawing test is subjective. 

To determine if the scores from the main scorer on the drawing test were reliable, both the main scorer and 
a second scorer graded responses from the same 20 participants. Responses from those participants that were 
assigned substantially different scores by the two raters were identified and discussed until a consensus on the score 
was reached. Although this exercise does not give us an estimate of reliability of the main rater’s scoring, it does go 
a long way in improving inter-rater reliability. We calculated the correlation between the scores from the two raters; 
this inter-rater reliability measure was found to be 0.97. 



Results 

We used spatial ability scores as the concomitant variable. One way ANCOVA analysis gave us the 
following F-values for the four dependent measures as well as the total scores. The adjusted means are also provided 
along with standard deviations (Table 1). 





Control 


Overview 


Prediction 


F values for one- 
way ANCOVA 


Drawing 
(out of 20) 


Mean = 17.84 
S.D. = 2.95 


Mean = 18.17 
S.D. = 1.80 


Mean = 18.68 
S.D. = 1.66 


F(2, 94)= 1.088 
p = 0.34 


Identincation 
(out of 20) 


Mean = 18.16 
S.D. = 1.90 


Mean = 18.94 
S.D. = 1.31 


Mean = 19.26 
S.D. = 1.12 


F (2, 94) = 4.653 

p = 0.01 


Terminology 
(out of 20) 


Mean = 16.28 
S.D. = 3.79 


Mean = 17.26 
S.D. = 1.98 


Mean = 18.00 
S.D. = 2.96 


F (2, 94) = 2.599 

p = 0.08 


Comprehension 
(out of 20) 


Mean = 14.56 
S.D. = 3.31 


Mean = 15.03 
S.D. = 3.03 


Mean = 15.86 
S.D. = 3.00 


F(2, 94)= 1.005 
p = 0.370 


Total 
(out of 80) 


Mean = 66.84 
S.D. = 10.32 


Mean = 68.81 
S.D. = 7.34 


Mean = 71.61 
S.D. = 6.81 


F (2, 94) = 2.606 

p = 0.080 



Table 1. Results 
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Discussion 



Clearly providing the students with static images of the three phases of the human heart and text describing 
these images (Overview condition) has an effect on comprehension. Asking students to make prediction based on 
this information (Prediction condition) had an additional but smaller effect. Although neither of these effects was 
significant for the total scores, the trend for the scores was in the direction expected - students in the Prediction 
condition performed better than both the Overview and Control conditions. These results are consistent with earlier 
research (Hegarty et ah, 2003). 
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Abstract 

This paper presents the results of a study to explore preserviee teaehers’ progress of their intent to use 
eomputer-based teehnology by taking instruetional teehnology integration eourse whieh mirrors the General 
Preparation of ISTE NETS profiles. This study explores how mueh the eourse makes preserviee teaehers have 
their higher intent to use eomputer-based teehnology and how important the instruetional teehnology integration 
eourse at the eollege is for preserviee teaehers. 

The instrument used in this study was the Intent to Use Computer-based Teehnology Survey (ITUCTS). 
The results showed that preserviee teaehers’ intent to use eomputer-based teehnology was signifieantly 
inereased after taking an instruetional teehnology integration eourse. 

Introduetion 

The researeher has been teaehing the eourse titled “Teaehing with Mieroeomputers” to preserviee 
teaehers at a major university in the Roeky Mountain region for several years. This eourse is a required eourse 
in the teaeher edueation program. The researeher observed that preserviee teaehers’ intent to integrate 
teehnology in elassroom is enhaneed after taking this eourse. In this study, the researeher aims to empirieally 
support his hypothesis. 

It is important for preserviee teaehers to experienee eomputer-based teehnology integration in eollege 
before they beeome teaehers so they have less anxiety and a positive level of intent to use eomputer-based 
teehnology in their elassroom in the future. In Sehrum’s (1999) researeh, more than 70% of edueation students 
were required to take three or more eredit hours of teehnology instruetion with an equivalent amount built into 
their traditional elasses. There are researehes that suggest that the more positive experienee preserviee teaehers 
have with eomputers, the less anxiety and the more positive level of intent they will have towards using 
instruetional teehnology (Downes, 1993; Koohang, 1989; Savenye, 1992). 

There have been many researehes that learners have positive attitudes towards using teehnologies in 
their elassroom. Using teehnology in edueational settings helps students in their learning. In addition, teaehers 
also improve their instruetion by using a variety of teehnology resourees sueh as the Internet, multimedia CD- 
ROMs, audio and graphies (Jao, 2001). There is no doubt that teaehing with teehnology provides more benefits 
for both teaehers and students than teaehing without any teehnology. 

Universities and eolleges are the plaees to train the preserviee teaehers to eomprehensively integrate 
instruetional teehnology into their future elassroom instruetion. It is neeessary for preserviee teaehers to be 
trained instruetional teehnology so that they ean use the teehnology skills in their future elassroom as teaehers. 

This researeh is to eompare preserviee teaehers’ intent to use eomputer-based teehnology before and 
after taking a “Teaehing with Mieroeomputers” elass at a major university in the Roeky Mountain region. This 
elass is an introduetory eourse to effeetive utilization of eomputers for instruetion; software/hardware seleetion; 
integrated applieations; databases; spreadsheets; word proeessing as applied to all areas of edueation. 

This eourse is based on General Preparation of the Teehnology Performanee Profiles for Teaeher 
Preparations of the International Soeiety for Teehnology in Edueation (ISTE) National Edueational Teehnology 
Standards (NETS). The NETS projeet writing team has ereated and defined teehnology performanee profiles for 
teaeher preparation aeross four eategories in hopes to use these profiles in the development and improvement of 
teaeher skills and eompeteneies in the area of edueation teehnology (SITE, 0000). The four profiles are General 
Preparation, Professional Preparation, Student Teaehing/Intemship, and First-year Teaehing. “This projeet, 
initiated by the ISTE, foeused on the development of national standards for various uses of teehnologies that are 
assoeiated with sehool improvement in the United States (Foley, 2001).” 
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The four profiles are not based on grade level. Instead they are based on the level of skill of people 
preparing for teaehing. The profile ealled General Preparation refers to students who have eompleted basie 
training in instruetional teehnology. ‘They may be at the lower division level or may have reeeived skill 
development through on-the-job training, obtaining a degree or experienee in a nontraditional program. . . They 
are able to perform the tasks that go beyond the elassroom setting” (ISTE, 2004). They ean operate teehnology 
systems, use eontent-speeifie tools sueh as software and graphing ealeulators, and use teehnology to eolleet 
information and proeess data. 

The Professional Preparation profile has eompleted intermediate training. “Experienees in this profile 
are part of professional edueation eoursework that may also inelude integrated field work” (ISTE, 2004). Before 
beginning student teaehing, students should able to perform at this profile. They identify the benefits of 
teehnology and resourees available, use hardware and software for speeifie teaehing and learning objeetives, and 
develop a portfolio of teehnology-based produets used to analyze teehnologieal resourees. 

The Student Teaehing/Intemship profile eontains those students who are finishing high level eourse 
work and internships to beeome teaehers. They are “eompleting their final student teaehing or intern teaehing 
experienee with extensive time spent with students. They are being supervised by a mentor or master teaeher on 
a eonsistent basis (ISTE, 2004). They ean solve problems with routine teehnology that oeeur in the elassroom, 
find teehnology resourees to support a eoherent lesson sequenee, and guide eollaborative learning aetivities to 
help students solve problems. 

The First-Year Teaehing profile means that they are ready to teaeh. “They have eompleted their formal 
teaeher preparation program and are in their first year of independent teaehing. They are typieally in eontrol of 
their own elassroom (ISTE, 2004). They should able to ehoose and arrange for aeeess to appropriate 
teehnology systems, resourees and serviees, and “plan and implement teehnology-based learning aetivities that 
promote student engagement in analysis, synthesis, interpretation, and ereation of original produets “(ISTE, 
2004). 

The purpose of the study is to explore the effeetiveness of instruetional teehnology integration eourse 
on preserviee teaehers’ intent to use eomputer-based teehnology in their future elassrooms. This researeh has the 
eomparison of using instruetional teehnology by preserviee teaehers based on their age, sex, and major between 
elementary and seeondary edueation. Answering these questions is important to future of edueation and to 
produee qualified teaehers in K-12 sehools. As teehnology inereasingly beeomes part of everyday life and all 
other endeavors, the ability of teaehers to teaeh using teehnology will affeet the lives and well being of all their 
students. 

Reeently, students want their teaehers to use appropriate teehnology in the elassroom to have better 
visual literaey, better than ehalkboard or whiteboard. Students sometimes have diffieulty reading their teaehers’ 
hand writing if their teaehers write on the ehalkboard or whiteboard like the researeher experieneed when he 
was in K-eollege. Teehnology is everywhere and ean support learning, and eomputers support eommunieations 
beyond elassroom walls (Shelly and three others, 2004). Teehnology ean help teaehers to improve their 
instruetion, to ehange the teaehing and learning proeess, to motivate students and to individualize students 
learning (U.S. Congress, 1995). 

The teaeher’ s role relates to what oeeurs in the elassroom and to what happens to students’ learning. 
Teaeher’ s attitudes toward the use of teehnology ean signifieantly affeet their students’ opportunities to learn 
about teehnology. In order to help K-12 students, training preserviee teaehers is the most direet and eost- 
effeetive way (Fasion, 1996). In the university setting, designing a eourse whieh ean effeetively train and 
enhanee preserviee teaehers’ intent to use teehnology is a direet and eost-effeetive way to transfer the 
applieation of teehnology from teaehers’ training program to pub lie sehools. 

Methodology 

This researeh aims to explore the effeetiveness of instruetional teehnology integration eourse on 
preserviee teaehers’ intent to use eomputer-based teehnology in their future elassrooms. In order to eolleet data 
from the partieipants, quantitative proeedures were used. This seetion ineludes a deseription of the sample, data 
eolleetion proeedures, instrumentation, and data analysis methods. 

Sample Deseription 

The partieipants in this study were preserviee teaehers who were enrolled in a “Teaehing with 
Mieroeomputers” elass at a major university in the Roeky Mountain region. This eourse mirrors the first General 
Preparation as per ISTE NETS profile. Therefore, the partieipants in this study represent preserviee teaehers 
who were taking the first instruetional teehnology eourse of the ISTE NETS General Preparation. The 
population for this researeh eonsisted of preserviee teaehers eurrently in teaeher edueation programs at a four 
year university and eurrently enrolled in a teehnology integration eourse. 
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The course has five sections which have a total of 125 students. The course is a required instructional 
technology courses for education majoring students. Since the participants were taking the course on campus, 
the researcher requested the instructors for their permission and then researcher administered the survey for the 
research in each classroom for all five sections. Data collection took place at the beginning and at the end of the 
Fall 2005 semester. 

Procedure 

Data were collected in the ‘Teaching with Microcomputers” courses which were taught by three 
different instructors. All three instructors of the course allowed the researcher to be in their classrooms for 
collecting data from the students enrolled in the course. The researcher distributed the questionnaires to each 
section during the first week of the semester for pre-survey and the last week of the semester for post-survey. 
The participation of subjects was voluntary in nature. Overall 94 students participated in the study, out of which 
61 were females and 33 were males, and 35 students had elementary education and 59 students had secondary 
education as their major. The age of participants was between 18 years and 46 years. 

Instruments 

For the study, one survey instrument and a demographic questionnaire were utilized to collect the data 
from the participants. They are described as follows: 

a) Intent to Use Computer-based Technology Survey (ITUCTS): The instrument used in this study 
was the Intent to Use Computer-based Technology Survey (ITUCTS). ITUCTS was adopted from the writings 
of Bichelmeyer, Reinhart, and Monson (1998) and Wang (2001). 

The ITUCTS is designed to measure beliefs about the teacher’s role in teaching with computer-based 
technology. The survey makes a distinction between teacher-centered and student-centered use of computer- 
based technology by focusing questions on the perceptions of the teacher’s role in a classroom with computer- 
based technology and the preservice teachers’ perceived use of computer-based technology in the classroom. 

The ITUCTS instrument is divided into two sections, each section has 12 questions. The first section 
addresses the preservice teachers’ perceptions of their future role in a classroom equipped with computer-based 
technology (Role). Role of the teacher in the classroom was defined as the manner or style in which the teacher 
engages during classroom instruction, having a spectrum from, the teacher as an authority figure (Teacher- 
Centered Role) to the teacher as a learning facilitator (Student-Centered Role). This section uses a Likert scale 
from (1) Strongly Disagree to (5) Strongly Agree with 12 questions. The second section addresses the preservice 
teachers’ perceptions of how they will use computer-based technology specifically when placed in a computer- 
based technology enhanced classroom (Use). Use of computer-based technologies in the classroom is defined as 
either the use of computer-based technology by the students for learning activities (Student-Centered Use) or 
use of computer-based technology by the teacher in ways that enable the teacher to more easily manage his or 
her classroom and instruction (Teacher-Centered Use). This second section also uses a Likert scale from (1) 
Never to (5) Frequently with 12 questions. 

The reliability of the section measuring teacher-centered role is .94, the section measuring student- 
centered role is .93, the section measuring teacher-centered computer use is .86, and the section measuring 
student-centered computer use is .93 (Wang, 2001). The overall reliability of this questionnaire in this study was 
found to be .83 and the reliabilities on the sub-scales were found to be similar to the study of Wang (2001). 

Data Analysis 

Data from ITUCTS and demographics questionnaire were organized in SPSS 13 statistical software to 
analyze. This study used group comparison analysis to determine the difference between preservice teachers’ 
intent to use computer-based technology before and after they take “Teaching with Microcomputer” course. 

Results 

The table below shows that there was a significant difference in mean scores of students in pre and 
post test on Intent to Use Computer-based Technology (Table 1). 
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Table 1: Mean Difference between pre and post scores on Intent to Use Computer-based Technology 





Mean 


S.D. 


T 


Pre 


87.7 


8.89 


5.21** 


Post 


92.5 


9.04 





^^significant at 0.05 level 



Further analysis showed that results were insignifieant on the role of Intent to Use Computer-based 
Teehnology Survey and its sub-seales. But results were signifieant for the use dimension of Intent to Use 
Computer-based Teehnology Survey and its sub-seales. 

The analysis showed that there was a signifieant differenee in mean seores of students in pre and post 
test on the use dimension of Intent to Use Computer-based Teehnology (Table 2). 



Table 2: Mean Difference between pre and post scores on Intent to Use Computer-based Technology (Use 
dimension) 





Mean 


S.D. 


T 


Pre 


42.7 


6.32 


5.74** 


Post 


46.5 


6.90 





**signifieant at 0.05 level 



The analysis showed that there was a signifieant differenee in mean seores of students in pre and post 
test on the teaeher eentered use dimension of Intent to Use Computer-based Teehnology (Table 3). 

Table 3: Mean Difference between pre and post scores on Intent to Use Computer-based Technology (T eacher 
Centered Use) 





Mean 


S.D. 


T 


Pre 


23.7 


3.23 


3.27** 


Post 


24.9 


3.10 





**signifieant at 0.05 level 



The analysis showed that there was a signifieant differenee in mean seores of students in pre and post 
test on the student eentered use dimension of Intent to Use Computer-based Teehnology (Table 4). 

Table 4: Mean Difference between pre and post scores on Intent to Use Computer-based Technology (Student 
Centered Use) 





Mean 


S.D. 


T 


Pre 


19.0 


3.93 


6.87** 


Post 


21.6 


4.35 




**signifieant at 0.05 level 









Limitations 

The limitation of the study is that the sample of the partieipants in the study is limited to one speeifie 
eourse and speeifie university. Henee, these results ean not be generalized throughout the eountry as the sample 
is not representative aeross USA. 

In the future researehers may improve and add to the results of this researeh by taking a more representative 
sample and eondueting the researeh in a more eontrolled setting. 
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Conclusion 



Schrum (1999) and Schrum, Skeele, and Grant (2002) state that education students should take three or 
more credit hours of technology instruction with an equivalent amount built into their traditional classes. There 
are researches that suggest that the more positive experience preservice teachers have with computers, the less 
anxiety and the more positive level of intent they will have towards using instructional technology (Downes, 
1993; Koohang, 1989; Savenye, 1992). 

This research proves that preservice teachers’ intent to use computer-based technology has been 
significantly improved after taking the instructional technology integration course. 
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In this session, the presenters will report survey findings related to the current status and future 
trends of blended learning in corporate settings across different cultures. Surveyed in this study 
are 568 practitioners in corporate training or e-learning in various workplace settings from four 
countries - i.e., Korea, Taiwan, the United Kingdom, and the United States. Findings of this 
survey study reveal which technologies and instructional strategies are currently used for blended 
learning in these different countries as well as the emerging technologies and trends surrounding 
the field of blended learning in workplace settings. Also, the perceptions of the respondents 
regarding the benefits of blended learning and the barriers to deliver it in their organizations are 
analyzed. 



With the emergence of Internet technologies, there has been an explosion of nontraditional learning opportunities 
during the past few years. This explosion is apparent in K-12 environments, higher education, and government and 
military training settings (Bonk & Graham, 2006; MacDonald & McAteer, 2003; Young, 2002). Such informal and 
nontraditional training approaches have also proliferated in corporate training (Cross, in press; Noe, 2003). However, 
various limitations of e-leaming as a training method in corporate settings have led many to try mixing various 
delivery methods. Accordingly, the interests in blended learning, which is typically combine face-to-face training 
and online learning, is rapidly increasing (Boyle, Bradley, Chalk, Jones, & Pickard, 2003; Duhaney, 2004; Thomson 
NETg, 2003; Thome, 2003). Millions of learners around the planet, in fact, are actually learning in this fashion each 
day (Bonk & Graham, 2006). Ironically, however, there is minimal known about the resulting learning differences 
among various blended learning models and the transfer of learning gains from one delivery mechanism (e.g., self- 
paced online learning to acquire content) and another (e.g., face-to-face classroom training to practice one’s new 
skills in front of others). 

Despite the many unknowns, blended learning estimates continue to climb. By the end of the decade, it is 
conceivable that 80-90 percent of college and corporate training classes will be blended (Kim, Bonk, & Zeng, 2005) 
and that more than one billions learners around the globe will be advancing their skills in this fashion. Therefore, 
many practitioners in HRD and corporate training find themselves in a situation where they consider catching on this 
latest trend of workplace learning but also they have many questions unanswered when it comes to blended learning. 
In response, we ask the question; where is blended learning actually headed? The present study intends to provide a 
compass that can mark the direction and intensity of the blended learning approach in corporate training settings. 
Blended learning is becoming a dominant delivery method for workplace learning across organizations within 
various sectors and of varying sizes. A recent survey indicates that the use of blended learning in all of training in 
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the United States will jump to nearly 30 percent during the next year, or about double that of 2004 (Balance 
Learning, 2004). Some are embracing blended learning as a training method that links learning and performance 
(Rossett, 2006). There are also expectations that blended learning can create more engaging learning environments 
(van Dam & Andrade, 2005) and help improve business performances (Harris, 2005). 

Although many organizations are recognizing the potential of blended learning to bring learning closer to employees, 
there are numerous issues to be addressed in delivering blended learning; in particular, in training sectors. First, 
there are a plethora of technologies and delivery methods that can be used for blended learning in training settings. 
Indeed, there are many different models and blended learning approaches for delivering training (Rossett, Doughs, 

& Frazee, 2003). Such a fact can lead to confusion for practitioners in deciding the optimal blended learning 
approaches for their organizations. Thus, practitioners need guidance on the effective and efficient methods for 
delivering blended learning. Second, although blended learning has been discussed in global perspectives in higher 
education settings (Bonk & Graham, 2006), such discussions have been lacking in training settings despite the fact 
that many researchers and practitioners have emphasized the importance and benefits of global collaboration in 
education and training. The present study addresses the first issue while exploring instructional strategies and 
emerging technologies related to the current status and future trends of blended learning in corporate training in five 
different countries. 

Clearly, a study of the future of blended learning and its implications for the delivery of learning for global learners 
is warranted. In response to this need, a survey was conducted of training professionals (e.g., chief learning officers, 
training managers, trainers/instructors, and e-leaming developers) from diverse cultures on the current status and 
future trends of blended learning in workplace learning settings. 

Review of Literature 

Blended learning has garnered a great deal of attention from both education and training settings around the world 
for several reasons. In education settings, particularly for higher education, blended learning has been recognized as 
an opportunity to improve the teaching and learning process by complementing the strengths and weakness of face- 
to-face and online learning settings. Blend traditional teaching approaches with learning technologies, thereby 
allowing for more interaction between instructor and students or among students than in face-to-face classroom 
instruction such as large class lectures (Chamberlain, Davis, & Kumar, 2005; Dziuban, Hartman, & Moskal, 2004). 
Blended learning can also allow increased accessibility and flexibility for classroom teaching, as well as increasing 
its cost-effectiveness, by reducing seat time in classrooms (Chamberlain, et ah, 2005; Dziuban, et al., 2004; 
Osguthorpe & Graham, 2003). Additionally, some blended learning approaches are adopted as a means to address 
the dissatisfaction of online students with the lack of sense of community in their online classes due, in large part, to 
the lack of face-to-face interactions (Lee & Im, 2006; Osguthorpe & Graham, 2003). 

Several studies have been conducted to explore the claimed effectiveness of blended classes with that of face-to-face 
or online classes. Results of several research studies of students in higher education settings suggest that student 
satisfaction and learning outcomes can be superior in blended learning settings to those in online settings (Boyle, 
Bradley, Chalk, Jones, & Pickard, 2003; Lee & Im, 2006; Lim, Morris, & Kupritz, 2006). Dzuiban and his 
colleagues (Dziuban, Hartman, Juge, Moskal, & Sorg, 2006) also report that the learning effectiveness of students 
enrolled in blended learning courses (i.e., students’ academic achievements and their withdrawal rates) are equal or 
superior to that of students in face-to-face or online courses. Furthermore, some studies suggest that faculty shows a 
high level of satisfaction with blended learning courses, due largely to increased flexibility and enhanced 
interactions in web-enhanced environments (Dziuban et al., 2004: Wingard, 2004). Additionally, a meta-analysis of 
the past studies on the effectiveness of web-based instruction compared to classroom instruction by Sitzmann and 
his colleagues (Sitzmann, Kraiger, Stewart, & Wisher, 2006) provides preliminary evidence of the effectiveness of 
blended learning programs by harnessing the effectiveness of the two different modes of instruction; i.e., online and 
face-to-face instruction. 

In addition to the benefits of blended learning reported in higher education settings, HRD and training professionals 
have touted the potential of blended learning for transferring learning in the workplace setting (Rosenberg, 2006; 
Rossett & Frazee, 2006; Shaw & Igneri, 2006). A study by Thomson NETg (2003) of learners in both education and 
training settings indicates that learner satisfaction and learning outcomes - i.e., level 1 and 2 evaluations in 
Kirkpatrick’s (1994) model for evaluating training programs - were higher in blended learning courses than in e- 
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learning courses. Another key benefit of blended learning in corporate training settings is an increase in the cost- 
effectiveness of course delivery by reducing the time and costs for employees to travel to participate in a classroom 
training (Bonk & Graham, 2006). A recent survey of learning professionals in the UK and the US also shows that a 
majority of learning professionals think that blended learning is the most efficient training method (Balance 
Learning, 2004). Some best practices and successful blended learning models in corporate training settings have 
been reported in the literature (Bersin, 2004; Bonk & Graham, 2006). It should be recognized, however, that 
empirical studies on blended learning in workplace learning settings are still scant. In effect, Shaw and Igneri (2006) 
contend that more research studies are needed to enhance our understanding of how blended learning can truly 
improve learning, performance, and retention. 



Methodology 

This survey was conducted of 568 employees from four different countries (i.e., Korea, Taiwan, Untied Kingdom, 
and the United States; additional data is still coming in from China). The figure below represents the breakdown of 
the total respondents by their location of employment (see Figure 1). The participants in this survey study belonged 
to various types of organizations, including government, business, and not-for-profit organizations. About 40 
percent of the respondents were female and 60 percent were male. This survey took place between November 2005 
and July 2006 using Survey Share, a Web-based survey tool. This survey is a part of a longitudinal study of the 
future of e-leaming in corporate training and higher education settings in 2003 and 2004 (Kim & Bonk, in press; 
Kim, Bonk, & Zeng, 2005). 



1. Where are you primarily employed? 
(N = 568) 



Taiwan 




24% 



Figure 1. Respondents’ location of employment 

The survey instrument for this study consisted of 3 1 items related to respondent demographics, current status of 
blended learning in the respondent organizations, and future predictions for blended learning in their organizations. 
Seven investigators, including three from Korea, two from mainland China, one from Taiwan, and one American, 
participated in developing the survey instrument. To address these international participants, the survey was created 
in four different languages (i.e., traditional and simplified Chinese, English, and Korean). The survey instrument 
was developed in English first, and then was translated into other languages by investigators who were speakers of 
the native language. The translation was then cross-checked by other investigators on our research team or by 
external colleagues to check for the accuracy of the translation and also for the validity of the instrument. The 
survey was distributed to several online forums and listservs for training and human resource professionals in the 
aforementioned countries. Some descriptive analyses (e.g., frequencies) were conducted of the data using a 
statistical analysis tool provided in the survey system used for this survey. 
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Findings 



The results of the present study indicate that blended learning has become a popular delivery mode in workplace 
learning settings. 65 percent of those surveyed responded that their organizations were already using blended 
learning approaches for training their employees and another 20 percent of them indicated that their organizations 
were considering using it at the time this survey was being conducted. This trend was similar across different 
countries surveyed in this study, while blended learning approaches were being used the least in Taiwan among the 
four participating countries. About 45 percent of the respondents from Taiwan reported their organizations were 
using blended learning approaches. Still, 30 percent of the Taiwanese respondents indicated they were considering 
using blended learning approaches, which is a higher percentage than that of any other countries being studied. In 
terms of the future state of blended learning, 68 percent of those surveyed predicted that their organizations’ budget 
spending on blended learning would increase during the next few years. Korean respondents were the most 
optimistic about their future training budgets on blended learning, with 84 percent of them saying their budget 
spending in blended learning is expected to increase in the new few years. 

What are key drivers for this increasing popularity of blended learning? A majority of respondents reported that 
improving the quality of the learning experience, an increase in the availability and accessibility of learning, and 
cost reductions were the major key drivers for adopting blended learning in their organizations. Additionally, nearly 
30 percent of those surveyed responded that an increase in the focus related to on-demand learning would promote 
blended learning the most in the new few years, followed by the blurring lines between work and learning (19%), an 
increase in the use of real world cases, stories, and examples in training (15%), and increasingly individualized or 
personalized learning (15%). 

In spite of the clear indications of the increasing importance of blended learning in the future of workplace learning, 
the results of this study also found that there were obstacles in adopting blended learning - mainly limited time and 
budgets. Interestingly, only 8 percent of the respondents recognized “lack of management support” as a problem of 
developing blended learning, which was usually viewed as a major challenge of delivering training. Our survey 
respondents also indicated that their lack of understanding of blended learning is the most important issue that 
needed to be addressed to implement blended learning successfully. This finding is important because 68 percent of 
respondents also indicated that blended learning was either important or very important for the strategic planning for 
training and development in their organizations for the coming years. Ironically, however, only less than a half of 
those surveyed (49 percent) answered that their strategic plans were addressing blended learning. This trend in the 
lack of strategic planning on blended learning was almost identical across the four different countries being studied. 

One of the most often asked questions that arises when delivering blended learning is what the optimal blends are 
(Rossett & Frazee, 2006). Figure 2 illustrates the results of this survey regarding instructional strategies that would 
be widely used in blended learning in the future. Our respondents predicted that instructional strategies that link 
learning and performance by providing learners with collaborative learning environments and authentic tasks will be 
used more often in the future. In contrast, didactic, lecture-based learning approaches and Socratic questioning were 
among the least favored. Clearly, Figure 2 reveals an emphasis on learner-centered, problem-based, and team-based 
approaches over instructor-centered ones. 
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Authentic cases and scenario learning 
Coaching and mentoring 
Problem-based learning 

Self-paced learning 
Virtual team collaboration and problem 
solving 

Simulations or gaming 
Guided learning 
Modeling of the solution process 
Exploration and discovery 
Discussion 

Lecturing and instructor-directed activities 
Debates and role play 
Socratic questioning 

(Number of Responses) 



□ 



0 
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Figure 2. Instructional strategies that are expected to be used widely for blended learning in the next few years. 



In another question, we listed 13 technologies and asked the respondents to pick a technology that was expected to 
be used most widely for blended learning in the future (see FigureS). Our survey respondents predicted that 
technologies that enable learners for just-in-time training or performance support, such as knowledge management 
tools and digital libraries or content repositories would be widely used in the future. The respondents also predicted 
that wireless and mobile technologies would be used widely for delivering blended learning. Interestingly, only a 
small number of respondents predicted that some collaborative learning tools, such as massive multiplayer online 
gaming, blogs, and wikis, would be used often in the future. This is a highly interesting finding given the exploding 
interest in such technologies in the media and in training related conferences and publications. This phenomenon is 
conceivably associated with corporate security restrictions which are extremely critical in workplace learning 
(Ardichvili, 2002). 



Knowledge Management Tools 
Digital Libraries and Content Repositories 
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Language Training and Support Tools 
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Figure 3. Emerging technologies that are expected to be used widely for blended learning in the next few years. 
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Another important question for delivering quality blended learning is how it will be evaluated. Survey respondents 
predicted that the quality of blended learning would be measured most often in relation to its benefits to their 
organizations such as employment performance, return on investment, or cost-benefit analysis (see Figure 4). It is 
notable that there is a trend toward evaluating blended learning at a higher level. The results show only 1 6 percent of 
the respondents’ organizations are evaluating blended learning at Level 3, improvement of behavior. However, 31 
percent of them projected that employee performance on the job will be the most effective criteria to measure the 
quality of blended learning in the future. Also, some respondents predicted that the effectiveness of blended learning 
would be best measured by comparing learner achievement in blended learning and classroom settings. Furthermore, 
only about 30 percent of respondents answered that their organizations were evaluating the quality of blended 
learning. Considering that 65 percent of those surveyed answered that their organizations were using blended 
learning approaches, less than a half of the organizations that were using blended learning approaches were actually 
evaluating them. 
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Figure 4. Respondents’ predictions on evaluation methods for blended learning in the coming decade. 

Conclusions and Implications 

In parallel with other survey studies (Balance Learning, 2004; eLearning Guild, 2003), the findings of the present 
study indicate that blended learning will become a popular delivery method in the future of workplace learning not 
only in western countries such as the United States and the UK but also in Asian countries such as Korea and 
Taiwan. Additionally, the results of this study shed light on where blended learning is headed in terms of the 
instructional strategies and emerging technologies that are expected to significantly impact the delivery of blended 
learning in the coming years. Despite the strong agreement among the respondents of this survey study regarding the 
increasing importance of blended learning for the future of workplace learning, they also indicated that they were 
facing several barriers to implementing blended learning in their organizations. One of the most noticeable barriers 
or issues that our survey respondents reported was their lack of understanding of the term blended learning. 

Apparently, there is a pressing need to provide training and HRD professionals with guidance on how to implement 
blended learning in their organizations. Consequently, the results of this study should help practitioners to become 
better informed of how blended learning will be designed, delivered, and evaluated for workplace learning in the 
future. Moreover, the findings of the present study on the current state of blended learning will provide some 
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direction for future researchers to address the issues that the training practitioners and managers are facing around 
the planet. Lastly, since there is scant research on blended learning that compares countries or regions of the world, 
the results of this study will provide meaningful data and ideas for serious decision making by both training 
practitioners and researchers related to blended learning around the globe. 

References 

Balance Learning. (2004, November). Blended is better. T+ D Magazine, 52-55. 

Bersin, J. (2004). The blended learning book: Best practices, proven methodologies, and lessons learned. San 
Francisco, CA: Pfeiffer Publishing. 

Bonk, C. J., & Graham, C. R. (2006). The handbook of blended learning: Global perspectives, local designs. San 
Francisco, CA: Pfeiffer Publishing. 

Boyle, T., Bradley, C., Chalk, P., Jones, R., & Pickard, R. (2003). Using blended learning to improve student 
success rates in learning to program. Journal of Educational Media, 28{Hy), 165-178. 

Chamberlain, M., Davis, L., & Kumar, M. (2005). The University of Houston embraces hybrid instruction. Texas 
Journal of Distance Learning, 2(1), 17-23. 

Cross, J. (in press). Informal learning. San Francisco, CA: Pfeiffer Publishing. 

Duhaney, C. D. (2004). Blended learning in education, training, and development. Performance Improvement 
Journal, 43(^\ 35-38. 

Dziuban, C., Hartman, J., Juge, F., Moskal, P., & Sorg, S. (2006). Blended learning enters the mainstream. In C. J. 
Bonk & C. R. Graham (Eds.), The handbook of blended learning: Global perspectives, local designs (pp. 
195-206). San Francisco, CA: Pfeiffer Publishing. 

Dziuban, C., Hartman, J., & Moskal, P. (2004). Blended learning. EDUCAUSE Center for Applied Research 
Research Bulletin. Boulder, CO: ECAR. 

eLearning Guild. (2003/ The blended learning best practices survey. Santa Rosa, CA: The eLearning Guild. 

Harris, P. (2005, November). Blended learning fuels sales at Toshiba. Learning Circuits. 

Kim, K.-J., & Bonk, C. J. (in press). The future of online teaching and learning in higher education: The survey 
says... EDUCAUSE Quarterly, 29(4). 

Kim, K.-J., Bonk, C. J., & Zeng, T. (2005, June). Surveying the future of workplace e-leaming: The rise of blending, 
interactivity, and authentic learning. eLearn Magazine. 

Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler. 

Lee, O., & Im, Y. (2006). The emergence of the cyber-university and blended learning in Korea. In C. J. Bonk & C. 
R. Graham (Eds.), The handbook of blended learning: Global perspectives, local designs (pp. 281-295). 

San Francisco, CA: Pfeiffer Publishing. 

Lim, D. H., Morris, M. L., & Kupritz, V. W. (2006). Online v^. blended learning: Differences in instructional 
outcomes and learner satisfaction. Paper presented at the Academy of Human Resource Development 
International Conference, Columbus, OH. 

MacDonald, J., & McAteer, E. (2003). New approaches to supporting students: Strategies for blended learning in 
distance and campus based environments. Journal of Educational Media, 25(2/3), 129-147. 

Noe, R. A. (2003). Employee training and development (3rd ed.). New York: McGraw-Hill. 

Osguthorpe, R. T. & Graham, C. R. (2003). Blended learning environments: Definitions and directions. Quarterly 
Review of Distance Education 4(3), 227-233. 

Rosenberg, M. J. (2006). Beyond e-learning: Approaches and technologies to enhance organizational knowledge, 
learning, and performance. San Francisco, CA: Pfeiffer Publishing. 

Rossett, A. (2006, January). Beyond the talk about blended learning. Chief Learning Officer. 

Rossett, A., Doughs, F., & Frazee, R. V. (2003, July). Strategies for building blended learning. Learning Circuits. 
Rossett, A., & Frazee, R. V. (2006). Blended learning opportunities. New York, NY: American Management 
Association. 

Shaw, S., & Igneri, N. (2006). Effectively implementing a blended learning approach: Maximizing advantages and 
eliminating disadvantages. Nashua, NH: Eedo. 

Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of web-based and 
classroom instruction: A meta-analysis. Personnel Psychology, 59(3), 623-664. 

Thomson NETg. (2003). Thomson job impact study: The next generation of corporate learning. Naperville, IL: 
Thomson NETg. 

Thome, K. (2003). Blended learning: How to integrate online & traditional learning. London: Kogan Page. 



182 




Young, J. R. (2002, March 22). ’’Hybrid” teaching seeks to end the divide between traditional and online instruction. 
Chronicle of Higher Education, p. A33. 

van Dam, N., & Andrade, M. (2005, April). Supercharge your blended learning. Chief Learning Officer. 

Wingard, R. G. (2004). Classroom teaching changes in web-enhanced courses: A multi-institutional study. 
EDUCAUSE Quarterly, 27(1), 26-35. 



183 




Motivational Influences in Self-Directed e-Learning 
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This presentation reports results of a mixed-method research study of working adults and college 
students who participated in self-directed e-learning courses on what influenced their motivation 
in their self-directed e-learning. The learner motivation is investigated in terms of what influences 
the learner motivation and whether the learner ’s motivational level changes during the instruction. 
Principles for the design of self-directed e-learning courses to sustain learner motivation are 
identified from this empirical study. 



The Web technology is changing the way people learn, work, and socialize (Bonk & King, 1998). More and more 
people are turning to the Web technology for their learning needs due to the flexible delivery system of the Web. 
Although the effectiveness of Web-based instruction has been proven in many studies (Jung & Rha, 2000; Olson & 
Wisher, 2002), high learner drop-out rates have been a concern in Web-based instruction (Carr, 2000; Cornell & 
Martin, 1997; Frankola, 2001; Islam, 2002), which also have been the case in distance education and computer- 
based instruction (Diaz, 2002; Moore & Kearsley, 1996). 

Past studies on the factors of learner attrition in distance education suggest that lack of time and lack of motivation 
are the major causes of that problem. Although instructional designers or instructors do not have control over the 
learner’s time, they can have some influence over learner motivation as it tends to change through instruction (Clark, 
2003; Coldeway, 1991; Song & Keller, 1999). Therefore, attention needs to be paid to improving learner motivation 
to address the issue of learner attrition in Web-based instruction. 

E-leaming is a rapidly growing market and is expected to be so in the future. A recent survey reported that the U.S. 
e-leaming market in 2002 was $10.3 billion (Adkins, 2002). It is projected that the U.S. e-leaming market will grow 
to $83.1 billion in 2006. Considering this large amount of spending on e-leaming, it is imperative that the 
investment to be worthwhile for the stakeholders. To this end, one needs to present a learning environment that 
builds success for online learners. Fostering adequate motivation for the online learner is one of the critical factors 
for creating a successful online learning environment (Hofmann, 2003). Yet, responding to the motivational 
requirements of online learners is a great challenge due to the lack of interaction in such learning environments 
(Bonk & Dennen, 2003; Cornell & Martin, 1997; Keller, 1999). 

Although the importance of learner motivation in Web-based instmction has been recognized in many literatures 
(Bonk, 2002; Ritchie & Hoffman, 1997), there is a lack of research on theories and practice of the design of 
motivating Web-based instmction (Keller, 1999; Song, 2000). Hence, there is a need to identify learners’ 
motivational needs to design motivating Web-based instmction. The purpose of this study is to investigate factors 
that influence learners’ motivation in Web-based instmction, in particular when they take self-directed e-leaming 
courses. Here self-directed e-leaming refers to a type of Web-based instmction in which the learner goes through 
instmction delivered through the Web in a self-paced format without the presence of an instmctor. In more detail, 
this study answers the following questions: 

• What influence the learner motivation in self-directed e-leaming? 

• Does the learner’s motivational level change during self-directed e-leaming? 

• What influences the learner’s motivation change during self-directed e-leaming? 

The results of this study are expected to increase our understanding of the motivational needs of the participants of 
self-directed online computer training by identifying what influences changes in their motivation to learn in a self- 
directed learning environment. The results of the study are expected to inform instmctional designers of how to 
design a motivating online learning environment. 
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Background of the Study 



This study investigated learners of self-directed online courses to answer aforementioned questions. Here, self- 
directed online courses refer to courses delivered via the Web in which learners go through instructional materials 
delivered via the Web at their own pace without the presence of an instructor. Adult learners can participate in 
online learning in various contexts, yet the self-directed online learning format is the focus of this study because 
self-directed online learning is a primary instructional format in training settings for adult learners (Driscoll, 2002; 
Galvin, 2002). 

The courses that the study participants took were offered by a major U.S. e-leaming vendor, who offers over 3,000 
online courses to 20 million learners per year worldwide. Those courses are offered to adult learners in various 
educational and workplace settings. The course format is stand-alone, typically 6-8 hours long, self-paced 
instruction delivered via the Web. The topics covered in those courses include desktop applications (e.g., Microsoft 
Office products), computer programming (e.g., JAVA, Oracle, MS .NET), soft skills development (e.g., coaching 
skills, consulting skills), and special topics tailored to the needs of specific organizations or fields. 

The learners participated in this study took self-directed online courses either in school or work settings. The 
learners in school settings took the online courses offered by the university either for personal development or as 
assigned by their course instructors. The learners in work settings also took the online courses either for personal 
development or to improve their job skills. 



Literature Review 

Motivation by definition is the degree of the choices people make and the degree of effort they will exert (Keller, 
1983). Past studies indicate that motivation is affected by affective, social, and cognitive factors (Relan, 1992). 
Keller (1983) identified four components of motivation - i.e., attention, relevance, confidence, and satisfaction - and 
strategies to design motivating instruction. Clark (1998) developed a CANE (Commitment And Necessary Effort) 
model that identified two processes of motivation: commitment and necessary effort. Wlodkowski (1993) suggests 
six major components that affect adult learners’ motivation in the time continuum. These motivational models were 
used in other research studies to identify the gap in learner motivation and how to design motivating instruction. 

Several theories have provided theoretical frameworks for understanding motivation (Pintrich & Schunk, 1996). 
Among different constructs on motivation, continuing motivation and intrinsic motivation are the most significant 
for instructional theory and research (Kinzie, 1990). Intrinsic motivation is defined as the motivation to engage in an 
activity “for its inherent satisfactions rather than for some separable consequence” (Ryan & Deci, 2000). Theories of 
motivation and empirical evidence have suggested several sources of intrinsic motivation. Some motivational 
researchers posit that activities that provide learners with a sense of control over their academic outcomes may 
enhance intrinsic motivation (Pintrich & Schunk, 1996). Lepper and Hodell (1989) have identified challenge, 
curiosity, control, and fantasy as primary characteristics of tasks that promote intrinsic motivation. 

Continuing motivation is the type of intrinsic motivation most directly concerned with education and it reflects an 
individual’s willingness to learn (Maeher, 1976). Studies have been done on how to improve learner motivation. 
Some theorists content that the primary reward for the learner is the activity itself; thus, continuing motivation is 
facilitated by an intrinsic interest in the activity (Condry & Chambers, 1978). Similarly, Merrill (2002) posits that 
the primary reward for the learner is learning itself - i.e., when the learner is able to show a new skill or an 
improvement in a skill, he is motivated to perform even better. He suggests it as an integration component of 
effective instruction. 

It is important to review past studies on motivational issues in computer-assisted instruction and distance education 
settings, since motivational features encountered these settings are similar to those in Web-based instruction (Song, 
2000). Kinzie (1990) argues that intrinsic and continuing motivation are important components in computer-based 
instruction. Malone (1981) suggests challenge, fantasy, and curiosity as the components of intrinsically motivating 
computer-based instruction. Song (2000) also argues that three types of motivation - motivation to initiate, 
motivation to persist, and motivation to continue - are important in Web-based instruction. Studies have been done 
on the effects of delivery medium to learner motivation. Several researches suggest that motivation to learn via a 
particular medium is influenced by the learner’s beliefs about his own ability and the difficulty level of the task, 
rather than by the medium per se (Clark, 1994). Similarly, Reinhart (1999) found that the learner’s self-efficacy and 
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task difficulty affects his motivation to learn via the Web. In addition, Keller (1999) posits that learner support is 
important for motivating learners in Web-based instruction. 

Methodology 



Participants 

The sample for the present study was drawn from the population of adult learners who had taken a self-directed e- 
leaming course in various education and training contexts. A sample of approximately 800 adult learners was 
selected from working adults and adult students across the United States, who were randomly selected from 
employees in the client organizations of the e-leaming vendor selected for this study, which is alluded to earlier. 
About 400 learners of self-directed e-leaming courses were selected from those who were undergraduate or graduate 
students enrolled in universities around the United States. Additionally, about 400 learners of self-directed e- 
leaming courses were also selected from working professionals in various workplace settings (e.g., business, non- 
profit, and government organizations). 

The Survey Instrument 

A survey instmment (i.e., a questionnaire) was constmcted to collect quantitative data in this study. A new survey 
instmment was developed by undergoing the following three steps to ensure its reliability and validity. First, a 
preliminary survey instmment was designed based on the theoretical framework that was developed from the review 
of literature, which was described in the earlier section. Second, the preliminary instmment was modified after a 
qualitative inquiry was completed prior to the present study in order to include additional motivational factors that 
were identified from the results of the qualitative research [see Kim (2004) for more information about the 
qualitative inquiry that preceded the present study]. Third, the survey instmment went through another modification 
phase after a pilot study was conducted to improve the reliability of the instmment. 

The resulting survey instmment has 60 questions, which comprises 59 multiple-choice questions and one open- 
ended question. This survey instmment is divided into three sections; (1) questions about the respondents’ 
backgrounds (questions 1-13), (2) Likert-type questions to measure the respondents’ perceptions of motivational 
influences in their self-directed e-leaming (question items 14-46), and (3) Likert-type questions regarding their 
motivation to persist in their self-directed e-leaming and to continue self-directed e-leaming in the future and an 
open-ended question for general comments about their perceptions of the self-directed online learning environment 
(questions 47-50). This survey instmment comprised of three a priori scales, which were conceptualized from the 
theoretical framework through the review of relevant literature (i.e., personal factors, internal factors, and external 
factors). 

Cronbach’s alpha was performed on this survey instmment with the data collected from the pilot study to measure 
internal consistency of the scales included in the instmment. The results of reliability analysis showed that the item 
on relevance had no strong correlation with any of the three a priori scales in the instmment. To address this issue, I 
added a new scale on relevance to this survey instmment by adding three more items related to relevance, which 
were adapted from a motivational questionnaire developed by John Keller, Instmctional Materials Motivation 
Survey (IMMS). By adding a new scale on relevance to this survey instmment, the final version of the instmment 
comprised of four a priori scales with the reliability coefficients ranging from .78 - .81 (see Table 1). The following 
table summarizes a priori scales in the survey instmment and the reliability of each scale. 



Table 1. A Priori Scales on Motivational Factors in the Survey Instmment 



Scale 


Question Items 


Reliability Coefficient (a) 


Personal Factors 


16, 22, 23,26. 27,31,33, 
40,41 


.79 


Internal Factors 


17, 18, 20, 28, 29, 34,35,38,39 


.80 


External & Social Factors 


21,24,30,32,36,37 


.78 


Relevance 


14, 27, 35, 44 


.81 



Data Collection and Analysis 

A message was sent to listservs or e-mail to the sample alluded to earlier to invite them to the survey. The message 
included information about the study and the URL to the survey site. As a result, a total of 368 individuals 
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completed the survey, yielding approximately a 46 percent response rate. The respondents took the surveys on the 
Web anonymously and their responses were stored on the Web server. The data was retrieved from the hosting 
server and was imported to a statistical analysis program. Descriptive statistics (e.g., means, standard deviations, and 
frequencies) and inferential statistics (i.e., Pearson’s chi-square, correlation analysis, factor analysis, and a 
multivariate analysis) were performed to analyze the data. 

Results 

Participants’ Demographics and Backgrounds in e-Learning 

Of 368 respondents to this survey, 43 percent took the e-leaming course as college students and 52 percent took the 
course as working professionals. The working professionals who participated in this survey belonged to 
organizations of various types: 45 percent belonged to a business organization, 39 percent belonged to a college or 
university, and 13 percent belonged to not-for-profit or government organizations. Among 158 college students who 
participated in this survey, 30 percent (n = 47) of them were undergraduate students and 70 percent (n = 1 1 1) of 
them were graduate students. Gender was equally distributed among the respondents to this survey: 46 percent were 
female and 54 percent were male. In terms of age, 18.2 percent of the respondents were 24 years old or younger, 

43.5 percent were between 25 and 34 years of age, 21.7 percent were between 35 and 44 years of age, and 16.6 
percent were 45 years old or more. Additionally, most of the survey respondents had moderate to high levels of 
experience in using computer and Internet technologies. Over 70 percent of respondents said they were using the 
Internet more than 20 hours per week, and 87 percent of them reported that they were using at least 3-5 software 
programs on a regular basis. 

Respondents to this survey study took self-directed e-leaming courses on various topics; 48 percent responded that 
they took a self-directed e-leaming course on desktop applications, 30 percent took a course on computer 
programming, and 22 percent took a course on soft skills. The time that respondents spent taking a self-directed e- 
leaming course also varied; 10 percent of those surveyed responded that they spent less than an hour taking the e- 
leaming course, 62 percent responded that they spent 1-6 hours in the course, and 19 percent responded that they 
spent 7 hours or more in the course. In terms of the respondents’ prior experience with online learning, 30 percent 
responded that they had had no prior online learning experience and the other 70 percent responded that they had 
taken 1-7 or more online courses, including college online courses and self-directed e-leaming courses. Results of 
the chi-square test revealed that the respondents in corporate training settings had more experience with online 
learning than did those in formal education settings [x2 (1, 366) = 12.770,/? < .05]. When asked about the frequency 
of their interaction with an instmctor or technical support staff, 3 1 percent of the respondents indicated that they 
never had such interactions and another 60 percent of those surveyed responded that they rarely or occasionally 
interacted with an instmctor or technical support staff. 

Factors Influencing Learner Motivation in Self-Directed e-Learning 

A factor analysis of the thirty-three Likert-scale items on motivational influences in self-directed e-leaming was 
performed using image factoring extraction method with varimax rotation. This factor analysis resulted in seven 
factors with initial eigenvalues over 1. A reliability analysis (i.e., Cronbach’s a) was conducted on each factor to test 
for internal consistency. Results of the reliability analyses revealed three factors had acceptable reliability levels, 
which ranged from .651 to .843 (see Table 2). 
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These items were negatively loaded on a factor and were reverse-coded when computing scale scores and 
reliability coefficients for each factor. 



Descriptive statistics were performed on the factors to investigate the mean scores of these factors. The mean scores 
of these three factors ranged from 3.02 to 3.73 on a 5-point scale (1 = “strongly disagree” and 5 = “strongly agree”), 
which suggests that the respondents’ had moderately strong belief that e-leaming was for them, or not for them, or 
they did not want to do it all by themselves (see Table 3). 
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Table 3. Descriptive Statistics for the Motivational Factors in Self-Directed e-Leaming 



Factors * 


N 


Minimum 


Maximum 


M 


SD 


Factor 1 


368 


1.67 


5.00 


3.4580 


.66509 


Factor 2 


368 


2.00 


5.00 


3.7310 


.46197 


Factor 3 


368 


1.00 


5.00 


3.0163 


.80438 



* Factor 1 = “E-leaming is not for me.” 

Factor 2 = “E-leaming is right for me.” 
Factor 3 = “I don’t want to be all by myself” 



Changes in the Learners’ Motivation during Self-Directed e-Learning 

The respondents’ self-reported overall initial motivational level was high (M = 4.95, SD = 1.272, where 1 = “very 
low” and 7 = “very high”) and their motivational level after they went through some lessons in the course was also 
high (M = 4.95, SD = 1.333, where 1 = “very low” and 7 = “very high”). When asked how their motivational levels 
changed as they went through the self-directed e-leaming course, those surveyed responded that their motivational 
levels remained the same (M = 3.01 and SD = .976, where 1 = “decreased significantly” and 5 = “increased 
significantly”), as shown in Figure 1. 



45 

|3§ 

° 30 
a: 25 
^ 20 
g 15 

$ 1 
“ 0 



- 




40.2 


- 


19.8 






- S.7 

















>1 
-Q s 
tU C 

a! iH 

c 

a “ 

CO 



m 

s 

Q 



23 



’D 

01 

Cfl 

m 



yw=3.09 
SD= 97S 



S.3 

HL 



■a 

01 



c 

B 

CO 



Figure 1. Results of Descriptive Analysis of Learner’s Motivational Changes during Self-Directed e-Leaming. 



Correlation analyses were performed to examine which variables were significantly correlated with learners’ 
motivational change. The results of correlation analysis showed that the learner’s motivational change - i.e. an 
increase or decrease in the learner’s motivation level during self-directed e-leaming - had a significant positive 
correlation with his or her satisfaction with the e-leaming course (r = .327. /? < .01) and with the frequency of his or 
her interaction with an instmctor or technical support staff during the self-directed e-leaming (r = .244,/? < .01). The 
learner’s motivational change also had a significant negative correlation with his or her age (r = -.107,/? < .05), as 
seen in Table 4. Among the three motivational factors identified from the factor analysis alluded to earlier, only the 
first motivational factor, “e-leaming is just for me” was significantly positively correlated with the learner’s 
motivational change (r = A61,p < .01). The other two motivational factors “e-leaming is not for me” and “I don’t 
want to be all by myself’ were not significantly correlated with the learner’s motivational change, where the 
correlation coefficients were -.031 {p = .548) and -.020 {p = .706) respectively. 



Table 4. Variables that are Significantly Correlated with the Learner’s Motivational Change during Self-Directed e- 
Leaming 



Variable 


Correlation Coefficient 
(r) 


Coefficient of 
Determination (r^) 


“E-leaming is right for me.” 


.467** 


.218 


Learner satisfaction 


.327** 


.107 


Interaction with an instmctor or 






technical support staff 


.244** 


.059 


Learner’s age 


-.107* 


.011 


*p<.05. **p<.0l. 
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Factors Associated with the Learner’s Motivational Change 

A multiple regression analysis was performed to identify the factors that were associated with learners’ motivational 
change during self-directed e-leaming. The dependent variable for this multiple regression analysis was the learner’s 
motivational change during self-directed e-leaming, where 1 = “decreased significantly” and 5 = “increased 
significantly”. Thirteen independent variables were entered for the stepwise regression analysis, which consisted of 
demographic variables (age, gender, respondents’ vocational status, and the setting in which they took the e-leaming 
course), the respondent’s backgrounds in e-leaming (computer competency, course topics, prior experience with e- 
leaming, time spent in taking the e-leaming course, and the amount of interaction with an instmctor or technical 
support staff in the self-directed e-leaming course), and the three motivational factors identified from the factor 
analysis. 

The results of this multiple regression analysis revealed that five out of thirteen variables entered for the stepwise 
multiple regression analysis significantly contributed to predicting the learner’s motivational change during self- 
directed e-leaming. Those five predictors accounted for 30.2 percent of the variance in the learner’s motivational 
change {F? = .302, adjusted 7?^ = .293). Table 5 summarizes the results of stepwise regression analysis, including 
regression coefficients, intercept, and adjusted R^. 

Table 5. Results of Stepwise Multiple Regression Analysis of Variables for Predicting the Learner’s Motivational 
Change during Self-Directed e-Leaming 





Unstandardized 

Coefficients 


Standardized 

Coefficients 






Adjusted 

R^ 




Variables ^ 


B 


SEB 


p 


Intercept 


R^ 


R 


1 


.988 


.098 


.467 


-.596 


.218 


.216 


.467 ** 


2 


.147 


.044 


.166 


-.592 


.241 


.236 


491 ** 


3 


.162 


.044 


.171 


-.667 


.268 


.262 


.518 ** 


4 


-.136 


.041 


-.146 


-.442 


.289 


.282 


.538 ** 


5 


.232 


.090 


.116 


-.488 


.302 


.293 


.550 ** 



Note. Formal education setting was the reference category for educational setting dummy variable, where the other 
category in this variable was corporate training setting. 

**;?< . 01 . 

§ 1 = Motivational factor: “e-leaming is right for me” 

2 = Learner satisfaction 

3 = Frequency of interaction with instmctor or technical support staff 

4 = Age 

5 = Educational setting. 

Furthermore, the results indicated that the first predictor (motivational factor) best explained the variability in the 
dependent variable, accounting for 22% of the variance {R^ = .218, adjusted R^ = .216). The second predictor 
(learner satisfaction) added 2.3% to the prediction [R = .491, F(2, 363) = 57.527,/? < .01] and 2.8% of the variance 
was increased by adding the third predictor (the frequency of interaction with an instmctor or technical support staff) 
to the equation [R = .518, F(3, 362) = 44.258,/? < .01]. The fourth predictor (age) added 2.1% to predicting the 
variance [R = .538, F (4, 361) = 36.160, p < .01], and the fifth predictor (educational setting) increased the 
prediction by 1.3% [R = .550, F (5, 360) = 31.193,/? < .01]. Additionally, the standardized regression coefficients 
(p) presented in Table 5 show that the fourth predictor (age) has a negative coefficient value whereas the other four 
predictors has positive coefficients values, indicating a negative linear regression between the learner’s age and his 
or her motivational change. 



Discussion and Conclusion 

The results of the present study have confirmed other research findings that learners’ motivational levels tend to 
change over time. Given the findings from this study that some learners experience a change in their motivational 
level during self-directed e-leaming, there are clear implications for designing motivating self-directed e-leaming 
environments. By understanding what is associated with the learner’s motivational change during self-directed e- 
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learning, e-leaming designers can be better informed on how to create a learning environment in which the learner is 
expected to sustain his or her motivation to learn during the self-directed e-leaming process. 



Although e-leaming designers cannot control some of the factors that were found to contribute significantly to 
predicting the learner’s motivational change during self-directed e-leaming, such as age and educational setting, 
they can indeed take other significant predictors into account when designing the learning environment to help 
learners stay motivated. Findings of this study suggest that the learner’s motivational level is likely to increase when 
the e-leaming course is designed in a way that is relevant to the learner, has multimedia components and hands-on 
activities, simulates real-world situations, provides feedback on the learner’s performance, and provides easy 
navigation on its course Web site. 

Furthermore, the findings of the present study indicate that in order for motivation to remain constant or increase, 
the e-leaming environment should be designed in a way that makes the learner satisfied with the overall learning 
experience and that provides him or her with opportunities to interact with an instmctor or support staff Interactions 
between the learner and instmctor or technical staff need to be considered especially when the e-leaming courses are 
designed or delivered for students in college or university settings, because such interactions are more likely to 
positively influence the motivation in these learners than in learners in workplace settings. 

The findings from the present study have implications for the motivational design of self-directed e-leaming courses. 
The nine factors that were found significantly associated with learner’s motivational change from the results of the 
present study can be translated into the principles for the design of motivating self-directed e-leaming courses. 

Figure 2 summarizes the instmctional design principles recommended for sustaining the learner’s motivation in self- 
directed e-leaming identified from the results of the present study. 



1 . Provide learners with content that is relevant and useful to them. 

2. Include multimedia presentations in the course that stimulate the learner’s interest. 

3. Include learning activities that simulate real-world situations. 

4. Provide learners with content that the difficulty level of which is just right for them. 

5. Provide learners with hands-on activities that engage them in learning. 

6. Provide learners with enough feedback on their performance. 

7. Design the Web site that is easy to navigate. 

8. Design the course in a way that the learner is satisfied with the overall learning experience. 

9. Incorporate some social interactions in the learning process (e.g., interaction with instmctor, 
technical support staff, or animated pedagogical agents). 

Figure 2. Instmctional Design Principles Recommended for Sustaining the Learner’s Motivation in Self-Directed e- 
Leaming 

It should be acknowledged that there are some limitations to this study. Since this study examined the self-directed 
e-leaming course format, it is likely that the findings of this study might be limited to this particular type of online 
learning environment. Therefore, readers should take caution not to generalize the findings of the present study to 
other types of online learning environments (e.g., instmctor-led online courses). Additionally, the present study 
focused on adult learners (e.g., adult students and working adults); it is possible that the motivational needs of adult 
learners might be different from those of school children or younger adults, as suggested by several theorists and 
researchers (Bohlin & Milheim, 1994; Gibbons & Wentworth, 2001; Wolcott & Burnham, 1991). Therefore, the 
findings of the present study may not be generalizable to learners of younger ages. 

It is possible that different factors might influence learner motivation in different types of online learning 
environments. Since this study was conducted with learners who took a particular type of online course, it is 
recommended that studies are conducted on what influences the motivation of learners in other types of online 
learning environments (e.g., instmctor-led e-leaming courses, and online courses in formal education settings). I 
suspect that such studies will reveal some motivational factors that were not found in the present study but have 
significant impacts in other online learning environments, and will also provide insights on whether the findings of 
the present study can be generalized to other types of online learning environments. 
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I also recommend that factors that influence the learner’s attrition from the e-leaming course be investigated in 
future research studies. Investigation of the factors that influenced learner attrition was not attempted in the present 
study as it was not the main focus of the study. But I believe that such a study will be beneficial in informing 
educators and instructional designers on how to design self-directed e-leaming environments that foster the learner’s 
motivation to persist in learning. Some studies have been done in Web-based distance education programs (e.g., 
Groleau, 2004; Tello, 2002), yet such research studies are lacking on self-directed e-leaming settings. Findings from 
such a study may provide one with an empirical basis upon which he or she can understand what factors prevent 
learners from persisting in self-directed e-leaming courses. 

References 

Adkins, S. S. (2002). The 2002 U.S. market for e-learning simulation. Retrieved August 26, 2003, from 
http://www.brandonhall.com/public/execsums/execsum_simmarket.pdf 
Bohlin, R. M., & Milheim, W. D. (1994). Applications of an adult motivational instructional design model. Paper 
presented at the Association for Educational Communications and Technology, Nashville, TN. 

Bonk, C. J. (2002). Online training in an online world. Bloomington, IN: CourseShare. 

Bonk, C. J., & Dennen, V. (2003). Frameworks for research, design, benchmarks, training, and pedagogy in Web- 
based distance education. In M. G. Moore & W. Anderson (Eds.), Handbook of distance education (pp. 
331-348). Mahwah, NJ: Lawrence Erlbaum Associates. 

Bonk, C. J., & King, K. S. (Eds.). (1998). Electronic collaborators: Learner-centered technologies for literacy, 
apprenticeship, and discourse. Mahwah, NJ: Lawrence Erlbaum Associates. 

Carr, S. (2000, Febmary 1 1). As distance education comes of age, the challenge Is keeping the students. The 
Chronicle of Higher Education, p. A39. 

Clark, R. C. (2003). Building expertise: Cognitive methods for training and performance improvement (2nd ed.). 

Silver Spring, CO: International Society for Performance Improvement. 

Clark, R. E. (1994). Assessment of distance teaching technology. In E. L. Baker & H. F. O'Neil, Jr. (Eds.), 

Technology assessment in education and training (pp. 63-78). Hillsdale, NJ: Lawrence Erlbaum Associates. 
Clark, R. E. (1998) Motivating performance: Part 1— Diagnosing and solving motivation problems. Performance 
Improvement, 37(8), 39-47. 

Coldeway, D. O. (1991). Patterns of behavior in individualized distance education courses. Research in Distance 
Education, 3(4), 6-10. 

Condry, J., & Chambers, J. (1978). Intrinsic motivation and the process of learning. In M. R. Lepper & D. Greene 

(Eds.), The hidden costs of reward: New perspectives on the psychology of human motivation. Hillsdale, NJ: 
Lawrence Erlbaum Associates. 

Cornell, R., & Martin, B. L. (1997). The role of motivation in Web-based instmction. In B. H. Khan (Ed.), Web- 
based instruction (pp. 93-100). Englewood Cliffs, NJ: Educational Technology Publications. 

Diaz, D. P. (2002). Online drop rates revisited. Technology Source (May/June). 

Dolezalek, H. (2004, October). The 23rd annual industry report. Training, 20-36. 

Driscoll, M. (2002). Web-based training: Creating e-learning experiences (2nd ed.). San Francisco, CA: Jossey- 
Bass/Pfeiffer. 

Frankola, K. (2001). Why online learners drop out. Workforce, 5^(10), 52-60. 

Galvin, T. (2002, October). 2002 industry report. Training, 24-73. 

Gibbons, H. S., & Wentworth, G. P. (2001). Androgogical and pedagogical training differences for online 

instmctors. Online Journal of Distance Learning Administration, 4(3). Retrieved July 12, 2003, from 
http://www.westga.edu/%7Edistance/ojdla/fall43/fall43.html 
Groleau, D. G. (2004). An analysis of barriers to adult learner persistence in online and face-to-face courses. 

Unpublished doctoral dissertation, Capella University, Minnesota. 

Hofmann, J. (2003, July 7). Building success for e-leamers. Learning Circuits. 

Islam, K. (2002, May). Is e-leaming floundering?: Identifying shortcomings and preparing for success. E-Learning 
Magazine. 

Jung, I., & Rha, I. (2000). Effectiveness and cost-effectiveness of online education: A review of the literature. 
Educational Technology, 40(4), 57-60. 

Keller, J. M. (1983). Motivational design of instmction. In C. M. Reigeluth (Ed.), Instructional-design theories and 
models: An overview of their current status (pp. 383-434). Mahwah, NJ: Lawrence Erlbaum Associates. 
Keller, J. M. (1999). Motivation in cyber learning environments. International Journal of Educational Technology, 
7(1), 7-30. 



192 




Kinzie, M. B. (1990). Requirements and benefits of effective interactive instruction: Learner control, self-regulation, 
and continuing motivation. Educational Technology Research and Development, 55(1), 5-21. 

Lepper, M. R., & Hodell, M. (1989). Intrinsic motivation in the classroom. In R. Ames & C. Ames (Eds.), Research 
on motivation in education. San Diego, CA: Academic Press. 

Maeher, M. L. (1976). Continuing motivation: An analysis of a seldom considered educational outcome. Review of 
Educational Research, 46(3), 443-462. 

Malone, T. W. (1981). Toward a theory of intrinsically-motivating instruction. Cognitive Science, 4, 333-369. 

Merrill, D. M. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43- 
59. 

Moore, M. G., & Kearsley, G. (1996). Distance education: A systems view. Belmont, CA: Wadsworth. 

Olson, T. M., & Wisher, R. A. (2002). The effectiveness of Web-based instruction: An initial inquiry. International 
Review of Research in Open and Distance Learning, 3(2). 

Pintrich, P. R., & Schunk, D. H. (1996). Motivation in education: Theory, research, and applications (2nd ed.). 
Upper Saddle River, NJ: Merrill Prentice Hall. 

Reinhart, J. (1999). Student motivation, self-efficacy and task difficulty in web-based instruction. Unpublished 
doctoral dissertation, Indiana University, Bloomington. 

Ritchie, D. C., & Hoffman, B. (1997). Incorporating instructional design principles with the World Wide Web. In B. 
H. Khan (Ed.), Web-based instruction (pp. 135-138). Englewood Cliffs, NJ: Educational Technology 
Publications. 

Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. 
Contemporary Educational Psychology, 25(1), 54-67. 

Song, S. H. (2000). Research issues of motivation in Web-based instruction. Quarterly Review of Distance 
Education, 1(3), 225-229. 

Song, S. H., & Keller, J. M. (1999). The ARCS model for developing motivationally-adaptive computer-assisted 

instruction. Paper presented at the Association for Educational Communications and Technology, Houston, 
TX. 

Tello, S. F. (2002). An analysis of the relationship between instructional interaction and student persistence in 
online education. Unpublished Ed.D. thesis. University of Massachusetts, Lowell. 

Wlodkowski, R. J. (1998). Enhancing adult motivation to learn: A comprehensive guide for teaching all adults (2nd 
ed.). San Francisco, CA: Jossey-Bass. 

Wolcott, L. L., & Burnham, B. R. (1991). Tapping into motivation: What adult learners find motivating about 
distance education. Paper presented at the 7th Annual Conference on Distance Teaching and Learning, 
Madison, WI. 



193 




Perils of “Teacher-proof’ Online Design: 

Study Findings from Science Web-based Inquiry Activities 

Violet A. Kulo 
Ward Mitchell Cates 
Lehigh University 

Having students become competent thinkers has long been an educational goal. Nickerson (1989) noted 
that many educators agree that we must help students learn to solve problems and think independently, creatively, 
and effectively (see also Bransford, Goldman, & Vye, 1991; Chipman & Segal, 1985; Resnick, 1987; Resnick & 
Klopfer, 1989). Bransford, Sherwood, Hasselbring, Kinzer, and Williams (1990) argued the basic problem is 
traditional instruction does not produce transfer to new problem-solving situations, while Bransford and Vye (1989) 
further contended that, “many traditional approaches to instruction do not help students make the transition from 
“knowing that” something is true to “knowing how” to think, learn, and solve problems” (p. 193). Studies in science 
education reveal that teachers depend heavily on textbooks in shaping their science curricula and instructional 
choices (Driscoll, Moallem, Dick, & Kirby, 1994; Kesidou & Roseman, 2002; Mitman, Mergendoller, & St. Claire, 
1987; Weiss, Banilower, McMahon, & Smith, 2001; Yore, 1991). In 1989, the American Association for the 
Advancement of Science (AAAS) indicted textbooks, saying. 

The present science textbooks and methods of instruction .... emphasize learning of answers more than the 
exploration of questions, memory at the expense of critical thought, bits and pieces of information instead 
of understandings in context, recitation over argument, reading in lieu of doing, (p. 14) 

In subsequent years, publishers have worked hard to improve textbooks and their accompanying teacher’s 
editions. Teacher’s editions provide scaffolds for learning to think pedagogically about particular content (Ball & 
Feiman-Nemser, 1988). These scaffolds are wrapped around the student pages right where the teacher needs them. 
The scaffolds provide comprehensive suggestions and activities that are designed to help teachers promote student 
learning, for example, lesson objectives, instructional strategies, common student misconceptions, specific 
questions, discussion topics, enrichment activities, and additional resources (Stodolsky, 1999; Woodward & Elliott, 
1990). 

Yet, according to the National Research Council (2000), traditional textbooks are not conducive to inquiry- 
based teaching. Calls for restructuring the way students learn in science recommend students be actively engaged in 
inquiries. Teaching and learning science by inquiry is the central tenet of the Benchmarks for Science Literacy 
(AAAS, 1993) and the National Science Education Standards (National Research Council [NRC], 1996). Hoffman 
(1997) suggested it might be possible, however, to use supplementary materials on the Web (see also NRC, 2000). 

In an attempt to investigate the extent to which the Web offers scientific inquiries for students, Bodzin, Cates, and 
Vollmer (2001) examined science sites drawn from The Eisenhower National Clearinghouse (ENC) Digital Dozen 
award list. Out of 209 award-winning science sites, they found only 19 sites (9.1%) contained inquiry activities. A 
subsequent study of 137 science sites found only 6 sites (4.4%) contained inquiry activities (Shive, Bodzin, & Cates, 
2004). According to Bodzin and Cates (2001), an activity must meet six criteria for it to qualify as a Web-based 
inquiry (WBI) activity: 

1 . A WBI must contain at least the first three essential features of classroom inquiry described in Inquiry and 
the National Science Education Standards (NRC, 2000): 

• Learners are engaged by scientifically oriented questions that may be stated implicitly or may be 
implied as a task. 

• Learners give priority to evidence, which allows them to draw conclusions and/or develop and evaluate 
explanations that address scientifically oriented questions. 

• Learners draw conclusions and/or formulate explanations from evidence to address scientifically 
oriented questions. 

2. The WBI must be learner-directed. The WBI should be phrased in such a way that learners would perceive 
it as directed at them. The majority of the wording used in the WBI should be directed at the learner 
(“you”), not at the teacher (“your students”). 

3. The WBI must support student learning of a science concept or content. Science WBIs must fall into a 
recognized science discipline (biology, chemistry, physics, environmental sciences, astronomy, 
oceanography and the like). 
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4. The WBI must be Web-based. A WBI is more than printed material placed on the Web, describing how an 
inquiry activity may be completed. Instead, it should go beyond simply being reformatted text from a 
printed sheet. It should be enhanced or customized to take advantage of the features of the Web to deliver 
instruction. 

5. Evidence used in a WBI should be of the same type an actual scientist would use. 

6. Conclusions and/or explanations must involve reasoning. They should be more than simple data analysis 
and reporting. 

These criteria focus almost exclusively on how WBIs meet student needs. Colburn (2000) contended, 
however, that the teacher is the key element in a successful inquiry-based classroom (see also Joyce, Weil, & 
Showers, 1992). In fact, the National Research Council (1996) states under Teaching Standard B that, “at all stages 
of inquiry, the teacher must guide, focus, challenge, and encourage student learning” (p. 33). If WBIs focus on 
students only, they might be mistaken for being a form of “teacher-proof ’ materials. According to Tanner and 
Tanner (1975), “teacher-proof’ materials are somewhat resistant to teacher tampering or to variations in individual 
teacher’s style, orientation, or competence, while Schubert, Schubert, Thomas, and Carroll (2002) defined “teacher- 
proof’ materials as, “materials that would achieve goals without distortion by teacher implementation” (p. 149). 

Pratt (1980) argued that in the 1960s designers of certain curricula decided teachers were not competent to teach the 
content. They developed curriculum materials that required little or no teacher intervention. Both content and 
methodology were packaged and accompanied by elaborate implementation instructions for teachers to follow 
(Schubert, 1986). The use of those materials did not, however, yield expected results in terms of increasing student 
achievement (Schubert, 1986; Tobin & Dawson, 1992). 

Much literature exists on the role teachers play in supporting student learning. Wright, Horn, and Sanders 
(1997) concluded the most important factor affecting student learning is the teacher (see also Brophy, 1986; Darling- 
Hammond & Bransford, 2005). Flick (1998) agreed and argued that, while a computer, textbook, or laboratory 
materials may serve as proxy for a “teacher,” the most important source of scaffolding in a classroom is the flesh- 
and-blood teacher (see also Krajcik, Blumenfeld, Marx, & Soloway, 2000; McKenzie, 2003). Crawford (2000) 
suggested inquiry-based teaching requires greater levels of involvement by teachers than in traditional teaching. In 
addition to the roles for the science teacher as motivator, diagnostician, guide, innovator, experimenter, and 
researcher articulated by Osborne and Freyberg (1985), Crawford proposed new roles that require more active and 
complex participation than simply being a facilitator or guide. Those roles are modeler, mentor, collaborator, and 
learner. Teachers, thus, make a difference and play an important role in making learning work. Soloway (1996) 
asserted that “teacher-proof curriculum is a silly, hostile notion, as is the idea that computer-based tutorials can 
replace teachers” (p. 11). 

According to Zion and Slezak (2005), the interaction of the self-directed student and the facilitating teacher 
promotes the inquiry process. Teachers, therefore, need to be knowledgeable in inquiry-based methods for students 
to understand inquiry. The National Research Council (2000) noted that most teachers have not had opportunities to 
learn science through inquiry or to conduct scientific inquiries themselves. A national survey of science and 
mathematics education found that 60% of K-12 science teachers reported they perceived a need for professional 
development in learning to use inquiry/investigation-oriented teaching strategies (Weiss, Banilower, McMahon, & 
Smith, 2001). Further, teachers implementing inquiry-oriented learning environments often lack appropriate and 
adequate instructional materials, both print and hands-on (NRC, 2000). Hammer (2000) posited that, to promote 
student inquiry, we must do more to understand and support teacher inquiry. As noted earlier, textbook publishers 
attempt to provide support for teachers through the scaffolds provided in the teacher’s edition. 

Project 2061, a science reform initiative of the American Association for the Advancement of Science 
(AAAS) developed a conceptual framework for analyzing how well science curriculum, instruction, and assessment 
help students achieve specific learning goals. That framework analyzes the extent to which a material’s content 
aligns with student learning goals; the extent to which the material’s instructional strategies support attainment of 
the specified learning goals; and the extent to which the material’s accompanying teacher’s edition supports teachers 
in helping students attain those specified learning goals (AAAS, 2000). Although the AAAS framework was 
designed for use in analyzing textbooks, we conjectured its instructional support criteria might be used to guide 
teachers in the selection of appropriate WBIs that will enable them to help students in achieving national 
benchmarks. 
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Purpose of the Study 



This study examined science WBIs to determine how well their features map to the AAAS instructional 
support framework. It sought to answer two questions: 1) What AAAS instructional support strategies are currently 
being employed in science WBIs and, 2) How well are those strategies being employed? 

Methodology 



Research Design 

This was a mixed methods study. Given the two questions, we employed content analysis (qualitative) and 
descriptive statistical analysis (quantitative). Content analysis makes “replicable and valid inferences from data to 
their context” (Krippendorff, 1980, p. 21). We chose this design because capturing data in their context is useful for 
identifying what exists and the way it manifests itself (Ritchie, 2003). According to Weber (1985), the best content 
analytic studies utilize both quantitative and qualitative methods. Miles and Huberman (1994) agreed and noted that 
during analysis of qualitative data, quantitative data can help to generalize specific observations, and one way this 
can be done is by “quantizing”; that is, counting qualitative information directly. Thus, we also used descriptive 
statistics to summarize and describe the data (Ary, Jacobs & Razavieh, 1990). 

Population and Sample 

We selected the 25 sites that had previously been validated by Bodzin and his colleagues as containing 
science Web-based inquiry activities as our population. They found 34 inquiry activities across 19 sites in the first 
study and 17 inquiry activities across six sites in the second study, yielding a population of 51 WBIs. We found one 
site had been duplicated across the two studies, six sites (12%) were no longer available or their locations had 
changed without redirecting us, and three sites were not science WBIs according to Bodzin and Cates’ (2001) 
qualifying criterion three. The non-science sites were mathematics and navigation, social studies, and language arts. 
This yielded a sample of 41 WBIs. 

Instrumentation 

To justify inferences, a content analyst must use an analytical construct that serves as the logical bridge 
between data and their context (Krippendorff, 1980). Using the AAAS instructional support framework as our 
analytical construct, we created an instrument that mapped observed functions and features of each WBI to 
identified AAAS strategies for enhancing student learning. The first two columns of the instrument contained AAAS 
strategies of instructional support. We added three columns: the first in which we mapped the observed feature to an 
identified AAAS strategy, the second in which we wrote the feature’s exact location (URL), and the third in which 
we wrote comments or analytical insights. 

Procedure 

A key issue in content analysis is validity. We randomly selected five sites using the www.random.org 
random number service. We tested the instrument through independent analysis of those five initial sites, using it to 
analyze how each previously validated WBI met the AAAS strategies for instructional support. Unfortunately, the 
initial AAAS instrument we used had criteria but no indicators of how to determine if something met those criteria. 
As a result, we had difficulty agreeing on how to rate the WBIs. Fortunately, the first author found another version 
of the AAAS instructional support framework whose criteria were further broken down into indicators that provided 
more guidance on how to analyze (AAAS, 2002). 

Using the new instrument, we reanalyzed our initial five sites. We discovered we still did not agree on all 
ratings and this necessitated discussion and resolution of our differences, including reanalysis of sites. We continued 
this interactive process until we agreed on our ratings and then the first author analyzed the entire set of 41 sites. 

Holsti (1969) emphasized that content analysis requires objectivity; that is, the analyst must use explicitly 
formulated procedures that minimize the possibility the findings reflect his or her subjective bias rather than the 
content under analysis. To confirm the objectivity of the first author’s ratings, we randomly selected five additional 
sites and analyzed them for inter-rater validation. Over several meetings, we met to compare results of our analysis, 
once again discussing differences, formulating rules to guide us, and reanalyzing until we were in agreement on all 
five sites. Following each of our interactive validation sessions, the first author re-examined all 41 sites to make sure 
her ratings continued to align with our validation discussions. Thus, in a total of 1 1 hours, we employed the multi- 
pass unanimous consensus technique described by Bodzin et al. (2001). 
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Findings 



To identify what AAAS strategies were employed in the sample sites and how well they were employed, 
we tabulated the frequency of each strategy. A summary of how the AAAS strategies were implemented across the 
sample sites is shown in Table 1. Strategies with a stronger implementation were employed in more than half of the 
sample sites; strategies with a moderate implementation were employed in at least half of the sample sites; strategies 
with a weaker implementation were employed in one quarter or less of the sample sites; and some strategies were 
not implemented at all. 

Table 1. Percentage of Strategies Implemented across Sample 



Status of Implementation 


# of Strategies 


%of74 


Stronger Implementation 


(51%-100%) 


25 


34% 


Moderate Implementation 


(26%-50%) 


16 


22% 


Weaker Implementation 


(l%-25%) 


24 


32% 


Not Implemented 


(0) 


9 


12% 



We counted the total number of AAAS strategies employed in individual WBIs and found they ranged 
between 15 and 44 strategies; that is, the site with the weakest implementation employed 15 strategies (20%) and the 
site with the strongest implementation employed 44 strategies (59%). Times to completion of examined WBIs 
varied. We completed the shortest activity ( http://whale.wheelock.edu/whalenet-stuff/Blubberglove.html ) in about 
half an hour while the longer activities, for example, http ://arcytech. org/i ava/population/ took approximately two- 
and-a-half to three hours to complete. We clustered our findings around six key concepts or practices: 

Concept: Learner Engagement 

This refers to whether the sites presented interesting and/or motivating activities to engage students. We 
found not all activities were engaging. For instance, the Probing your Surroundings activity 
( http://wise.berkelev.edu/student/topFrame.php?proiectID=5358 ) involved measuring the temperature of objects 
around the classroom. That activity seems commonplace and unlikely to engage most students. Some sites pretended 
to present activities about bigger issues. For example, the description of the Keeping an Eye on Ozone activity 
(http://pathfinderscience.net/ozone/index.cfm) was about how ground-level ozone can affect the respiratory system 
and is harmful to very young and elderly people and those people with chronic lung disease. Yet the activity 
involved measuring the level of ground-level ozone and its effect on common milkweed leaves. Other activities 
seem likely to have little personal relevance to most students. For instance, while the Malaria Introduction activity 
( http://wise.berkelev.edu/student/topFrame.php?proiectID=4439 ) may appeal to students who live in places with 
tropical climates where the problem is prevalent (like Africa and South America), that same activity may not be 
equally interesting and/or motivating to students who live in cold areas (like Wyoming or North Dakota) where 
malaria is not a problem. Conversely, urban sprawl and AIDS are universal and relevant problems and so the Sprawl 
in the Lehigh River Watershed activity (http://www.leo.lehigh.edu/envirosci/enviroissue/sprawl/index.html ) may 
appeal to a wide range of students regardless of their location. 

Concept: On-going Learning 

This refers to whether the sites connected instruction across activities. We found the examined WBIs 
existed in isolation as single activities; they were not presented in a scaffolded sequence that would ensure on-going 
learning. Activities consisted mainly of filling out worksheets, data collection and submission, doing experiments, 
filling out journals, debates, and online discussions. Activities that involve experiments or data collection and 
submission do not lend themselves well to being in a scaffolded sequence because once the step-by-step experiment 
is complete or the data are submitted, that marks the end of the activity. Activities that involve debates, online 
discussions, filling out journals, however, routinely encourage students to express their ideas. 

Concept: Learning over Time 

It appears AAAS intended for learning to be spaced out over time. Support for this is evident in their use of 
words that imply a timed learning sequence such as routinely, sequences, periodically, infrequently, and 
consistently. We found not all activities were spaced out to allow for a timed learning sequence. Spaced activities 
relate individual activities to one another in which students and teachers behave consistently in specified ways. For 
instance, most of the CIESE WBIs which make up 15% of the sample sites have students collect data, fill out 
worksheets, and collaborate with peers and experts. Teachers, on the other hand, are expected to facilitate 
implementation of those inquiries and assess students. Ten sites (24%) had one activity each and would likely be 
completed in one class period. Fourteen sites (34%) had multiple activities, while 17 sites (42%) had one activity 
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each that required students to collect data or do an experiment over a period of days. Thus, approximately three 
quarters of the WBIs examined appeared likely to take more than one class period to complete and would have 
provided opportunities for students to conduct them over a period of time. 

Concept: Task Novelty 

This refers to whether the sites included novel tasks and assessment to judge transfer of what has been 
learned. We had trouble interpreting what AAAS means by novel. One way to define novel is non-traditional; that is, 
activities that are not commonly done in the “traditional” classroom. We found activities designed with the same 
model had the same nature of tasks and assessment. For instance, activities designed with the CIESE model involved 
filling out worksheets and collecting data; activities designed with the WISE model involved, among others, online 
discussions and filling out journals; the Carolina Coastal Science activities presented resources on different 
problems for students to explore, choose a stakeholder position, and debate on how they would vote on those 
problems while justifying their position; the PathFinder Science activities involved doing experiments, collecting 
and submitting data. If one would classify filling out worksheets, doing experiments, filling out journals, collecting 
and submitting data as activities commonly done in the “traditional” classroom then they would not qualify as novel. 
Twenty-nine sites (71%) contained activities that were not novel, whereas 12 sites (29%) contained novel activities 
such as debates and online discussions. 

Practice: Addressed Prerequisites 

This refers to whether the sites adequately addressed prerequisite ideas and skills, and made adequate 
connections between specific ideas and their prerequisites. In the classroom, teachers often help learners to recall 
prerequisites and they make clear how those prerequisites connect to the next lesson. We found most WBIs stand 
alone and are not presented in a unit that contains related activities in which connections between ideas and their 
prerequisites can be made. Some WBIs presented background information on the activities which seemed to be 
content related to the topic under study but did not provide adequate prerequisite knowledge and skills that learners 
would need to conduct the inquiries. 

Practice: Support for Teachers 

The American Association for the Advancement of Science intended for materials to provide a lot of 
support for teachers but we found most sites had little support for teachers and some sites did not provide any 
support at all. Table 2 presents the distribution of teacher-support strategies across the sample sites. Twenty-four 
sites (59%) across the sample had teacher sections. The 19 teacher-support strategies in Table 2 were employed in 
less than half of the sample sites. The teacher-support strategy that was employed the most in 18 sites was 
recommending resources for enrichment activities and for improving the teacher’s understanding of key ideas. 

Table 2. Percent of Teacher Support Strategies across Sample 



Strategies 


# WBIs Found 


%of41 


Recommend resources 


18 


44% 


Assist in identifying students’ ideas 


13 


32% 


Suggest encouraging students’ questions 


12 


29% 


Suggest engaging students monitor change in ideas 


8 


20% 


Provide sufficiently detailed answers 


8 


20% 


Provide specific feedback 


6 


15% 


Alert to prerequisites 


5 


12% 


Interpret responses 


3 


7% 


Alert to specific ideas for which prerequisites needed 


2 


5% 


Alert to commonly held student ideas 


2 


5% 


Instruct to carry out expected performance 


1 


2% 


Alert to how ideas have been simplified 


1 


2% 


Probe beneath initial responses 


0 


0 


Suggest taking into account own students’ ideas 


0 


0 


Diagnose student errors 


0 


0 


Suggest using assessment for instructional decisions 


0 


0 


Provide interaction examples 


0 


0 


Modify activities 


0 


0 


Validate students’ experiences 


0 


0 
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In the classroom, teachers tend to recommend additional resources after the lesson to help learners further 
their understanding. Other interactions such as addressing common student misconceptions, diagnosing student 
errors, providing feedback, interpreting responses, probing students’ ideas, and elaborating concepts are likely to be 
done during the lesson. Those scaffolds that are likely to help the teacher during the inquiry process were not as well 
implemented as recommending enrichment activities which the teacher would, ideally, need after students conduct 
the core activity. It is worth noting that out of the nine instructional support strategies that were not implemented at 
all in any of the sample sites, seven were teacher-support strategies. Kinnaman (1993) asserted that far too much 
effort still goes into trying to make technology “teacher-proof’; yet, technology possesses no inherent philosophy of 
education, nor can it independently participate in the art of teaching. 

Discussion 

We found that 6 sites of the 50 previously validated WBIs (12%) were no longer available or their locations 
had changed without redirecting us. It is unclear why one out of eight sites has disappeared but one might conjecture 
that some sites may have lost the funding that supported them. It is also possible that the attrition may be due to 
limited staff to maintain the sites. Of the examined WBIs, 24 sites (59%) are hosted on university, college, or high 
school Web sites; 15 sites (37%) are designed and sponsored by science organizations; one site (2%) is maintained 
by a government organization; and one site (2%) named a single individual as the creator of the site. Universities, 
colleges, schools, government organizations, and science organizations are more likely to be continually funded and 
so WBIs supported and maintained by those institutions may be more likely to continue to exist. 

This study found that not all activities were engaging. Manner and Cates (in press) suggested students will 
engage with inquiry if they feel they are contributing to solving a real, local, and relevant problem. Harmer and 
Cates illustrated the student’s sense of the relevance of the problem with a Hierarchy of Empowerment, from the 
lowest sense of personal relevance to the highest sense of the value of deriving a solution. It might be challenging 
for the people designing WBIs to customize them for personal relevance since activities on the World Wide Web are 
by nature available globally. Any one of the WBIs may be local and relevant for one group of students and not 
another, depending on where the students are located. A few sites pretended to have greater significance than it 
appears the activity warranted. This might be an acknowledgement by designers that learners need to engage with 
inquiry, and telling them upfront the problem is important might engage them. While this might create initial 
interest, learners are likely to disengage once they realize the true importance of the inquiry. 

Although standalone WBIs may be useful, they may not meet AAAS criteria for instructional support. The 
reason for the lack of a scaffolded sequence may be due to the fact that the sample sites exist in isolation; they are 
not integrated in a curriculum and so instruction is not connected. Textbooks publishers often correlate the content, 
scope, and sequence in textbooks with published curricula of states and school districts (Squire & Morgan, 1990). In 
contrast, WBIs tend to be developed as standalones, perhaps by science organizations, professors, teachers, and 
scientists who have interest in a particular content or topic. Clearly, single activities do not lend themselves well to a 
scaffolded sequence since a sequence requires more than one activity. Even if existing WBIs could be put in a 
scaffolded sequence, someone would need to create that sequence. Historically, no organization to do such 
sequencing has existed. 

Teachers have not been well addressed in examined WBIs, certainly not to the extend AAAS believes they 
need to be. Support for teachers across the examined sites was usually background information related to the topic or 
non-scaffolded enrichment activities. There could be several reasons for this. First, WBIs may not be focused on 
teachers because they are learner-directed, which is criterion two an activity must meet for it to qualify as a WBI 
(Bodzin & Cates, 2001). Second, designers of WBIs may be talented at conducting inquiries but may not realize 
they need to instruct teachers what to do. Bodzin and his colleagues might have gotten half of the picture of what 
currently exists because they focused on the extent to which the Web offers scientific inquiries for students. They 
did not focus on the whole inquiry process but on the form such inquiries take. It might be possible that since all 
three of the designers of the original WBIs criteria (Bodzin, Cates, & Vollmer, 2001) are experienced teachers and 
have experience with inquiries, they may have assumed teachers know what to do and thought there is no need to 
tell them what to do. This could be a blind spot for designers who might assume that users of instructional materials 
will know what to do. Third, it may be challenging for designers to design scaffolding. Inquiry produces different 
results; it does not have a repeated experience. Designers may, therefore, have difficulty anticipating what students 
need because those needs may be unpredictable. 
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Recommendations 



To address the problem of attrition and help keep sites alive, one might be able to create a central site, for 
example, a clearinghouse that would house new WBIs and link to existing ones. Another role of the clearinghouse 
would be to improve and consolidate WBIs to fit into a larger context. An illustration of the logic of that central site 
is presented in Figure 1. The columns on the left of the illustration denote the process one would use to select a 
WBI. Users would enter the site by logging in through the portal and identify their location to bring up activities that 
are real, local, and relevant to them. That would solve the problem of learner engagement since users would be able 
to select problems that are local and relevant to them. The clearinghouse could integrate WBIs in the standard 
curriculum driven by national benchmarks and standards by mapping a list of all AAAS informed WBIs to popular 
textbooks. The clearinghouse could cluster WBIs contextually and put them in a scaffolded sequence in which 
prerequisites of each activity are connected to the assessment of the preceding activity. That would help address the 
problem of on-going learning. The clearinghouse could build scaffolds around WBIs to help solve the problem of 
lack of support for teachers. Users would select the desired textbook, chapter, and a WBI with its associated 
scaffolds. The clearinghouse could develop an instructional model with guidelines like the model used in Web-based 
Inquiry Science Environment (WISE) or PathFinder Science activities. The clearinghouse would validate a site if it 
had all components of the model and then would recognize it publicly. 

Figure 1. Illustration of the Logic of Suggested Clearinghouse 
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But such a clearinghouse would require staff and funding. One way the clearinghouse could be funded 
would be by a seed grant from the National Science Foundation or a similar agency. Another option might be that 
textbook publishers could underwrite the clearinghouse and be able to link to it from their Web sites. It could be 
staffed with multiple experts like curriculum designers, teachers, instructional designers, and Web designers to 
design new WBIs and link to existing ones. The clearinghouse could also build a community of practice where 
teachers discuss classroom practice and share their own experiences, techniques and approaches to the inquiry 
process. In addition, it may be possible to map those WBIs to higher education textbooks. This would help pre- 
service science teachers learn science through inquiry and, thus, address the problem of most teachers not knowing 
how to conduct scientific inquiries. 
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Abstract 

Using a quasi- experimental design the researchers investigated the comparative effects of individually- 
generated and collaboratively-generated computer-based concept mapping on middle school science concept 
learning. One hundred sixty-one students (74 boys and 87 girls) in eight, seventh grade science classes at a middle 
school completed the entire study. Qualitative data were analyzed to explain quantitative findings. The researchers 
concluded that the concept mapping software, Inspiration™, fostered construction of students ’ concept maps 
individually or collaboratively for science learning and helped students capture their evolving creative ideas and 
organize them for meaningful learning. 



Introduction 

According to Novak (1998), concept mapping is a process of organizing and representing concepts and 
their relationships in visual form. Concept mapping is one tool that can overtly engage students in meaningful 
learning processes. Further, concept mapping promotes meaningful learning and retention of knowledge for long 
periods of time and helps students negotiate meaning (Hyerle, 2000; Novak, 1990). 

McKenzie (1998) said that graphic organizers such as concept maps or thinking maps allow students to 
convert complex and messy information into meaningful displays. A variety of research studies have demonstrated 
the positive value of using concept mapping in the science classroom (Boujaoude & Attieh, 2003; Boxtel, van der 
Linden, Roelofs, & Erkens, 2002; Fischer, Bruhn, Grasel, & Mandl, 2002; Madrazo & Jordi, 2002; Odom & Kelly, 
2001; Okebukola & Jegede, 1988; Roth, 1994; Snead & Snead, 2004). Concept maps are a useful science education 
tool for engaging students’ understanding (Jonassen, 2000). 

This present study builds on previous research on the effect of the use of visualization during study time 
(Cifuentes & Hsieh, 2003a, 2003b) and the effect of student constructed visualization using computers as a cognitive 
tool (Cifuentes & Hsieh, 2004; Hsieh & Cifuentes, 2006). Visual representation facilitates both recall and 
comprehension (Cifuentes & Hsieh, 2003a, 2003b; Gobert & Clement, 1999; Hsieh & Cifuentes, 2006). 

The purpose of this study was to investigate the comparative effects of individually or collaboratively 
generated computer-based concept mapping on middle school science learning and quality of concept maps. Using a 
quasi-experimental design, the comparative effect of individually and collaboratively generated computer-based 
concept mapping on science concept learning was investigated. The researchers asked the following research 
questions: 

1 . Do middle school students who collaboratively or individually generate computer-based concept maps 

perform better on a comprehension test than those who do not generate computer-based concept maps? 

2. Do middle school students who collaboratively generate computer-based concept maps perform better on 

a comprehension test than those who individually generate computer-based concept maps? 

3. Will the quality of concept maps generated collaboratively exceed the quality of concept maps generated 

individually? 

Additionally, answers to the following questions will help explain results: 

4. How did students’ attitudes toward generating concept maps during study time differ across the 

individually-generated concept mapping group and the collaboratively-generated concept mapping 

group? 

5. What specific learning strategies were used in each group to prepare for the test and did they differ 

according to group? 

Method 

A non-randomized, control-treatment group, post-test only, quasi-experimental design was used to 
investigate the relative effect of individually or collaboratively generated computer-based concept mapping on 7th 
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grade science concept learning. Using both quantitative and qualitative methods helped triangulate results from 
diverse data sources. 

Participants 

One hundred and sixty one students (74 boys and 87 girls) in eight 7^^ grade science classes at a middle 
school completed the entire study. Using prior science performance scores to assure equivalence of student 
achievement across groups, the researchers assigned the teacher’s classes to one of the three experimental groups. 
The three experimental groups were equivalent across students’ academic performance and prior knowledge of 
science content. 

Design, Procedures, and Instruments 

The independent variables were gender and treatment group. The treatment group consisted of three levels: 
the control group of 40 students, 59 students trained to individually generate concept maps on computers, and 62 
students trained to collaboratively generate concept maps on computers. The dependent variables were science 
concept learning as demonstrated by comprehension test scores and quality of concept maps as demonstrated by 
rubric scores. 

Computer-Based Concept Mapping Workshop Training 

Both experimental groups (individual and collaborative) that created concept maps on computers while 
studying the science concepts attended three days of workshops on computer-based concept mapping of science 
concepts on computers in their computer laboratory before starting to study science concepts. These topics were not 
related to topics tested after experimental treatments. 

Instructional Format of Treatment Across Groups 

The control group, the individually-generated concept mapping group, and the collaboratively-generated 
concept mapping group had equivalent learning experiences for fifty minutes during five days, except that students 
in the control group worked independently without being told to apply a particular study strategy, students in the 
individual group worked individually to construct concept maps, and students in the collaborative group worked 
collaboratively to construct concept maps during their study time. Students in the individual and collaborative 
experimental groups first learned how to develop concept maps on computers using Inspiration™^ were then given 
science concepts to study using concept mapping, and were then tested on those concepts. 

The comprehension test. 

The comprehension test consisted of fifty paper-and-pencil based multiple-choice items from the Glencoe 
Science/ Texas Science for grade textbook (Biggs, Feather, Snyder, & Zike, 2002) that was provided with the 
seventh grade textbook adopted by the participating school district (^.86). Items were criterion referenced to 
concepts that students studied during their experimental study time. 

Learning strategy questionnaire and computer use survey. 

All participants were asked to fill out a ’’Learning Strategy Questionnaire” in the last minutes of the fifth 
day. The ’’Learning Strategy Questionnaire” is a student self-report instrument developed by the researcher. 
Students were asked to determine whether or not they had been exposed to the reading content prior to studying. 
Also, students were asked to explain how they felt about making concept maps that show interrelationships among 
concepts during study time and discuss how making concept maps helped them learn content. All students 
completed the ’’Computer Use Survey” administered to investigate students’ use of computers at school and at home 
as well as the frequency of using computers to support a number of purposeful tasks. 

Rubric for assessing the quality of concept maps. 

Concept maps generated by students individually and collaboratively during five days of treatment and 
testing were analyzed for quality by three reviewers (two teachers and one researcher). Concept maps were scored 
according to the four scoring components created by Novak and Gowin (1984). Reliability was established by 
obtaining three independent scores on each concept map. Three reviewers independently scored students’ concept 
maps as the final products of the five day experiment based on the rubric formula created by Novak and Gowin 
(1984). Inter-rater reliability was computed by determining the correlation between three reviewers and scoring for 
each concept map. The inter-rater reliability correlations between reviewer A and B, A and C, and B and C using 
Pearson’s Correlation were .97**, .98**, and.98**, indicating significant correlations at the 0.01 level. 
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Data Sources 

Data sources included (a) students’ comprehension test scores, (b) students’ responses to the “Learning 
Strategy Questionnaire,” (c) students’ responses to the “Computer Use Survey,” (d) rubric scores for assessing the 
quality of concept maps, (e) students’ study notes, (f) students’ concept maps, and (g) the researcher’s reflective 
journal that documents observations of students’ learning behaviors during the workshop and experiment. 

Results 

Results of data analyses provided answers to the research questions regarding the effect of individually or 
collaboratively generating concept maps on comprehension, effects of individually or collaboratively generating 
concept maps on quality of concept maps, students’ attitudes toward generating concept maps, and learning 
strategies used by students across groups. 

Effects of Individually or Collaboratively Generating Concept Maps on Comprehension 

An one-way analysis of variance (ANOVA) applied using treatment group as the independent variable and 
comprehension test scores as the dependent variable revealed that means differed across groups (see Table 1). The 
two experimental groups’ mean scores were respectively: individual group {n = 59, mean = 29.10, SD = 8.85), and 
collaborative group {n = 62, mean = 31.39, SD = 8.46). The control group {n = 40) yielded a significantly lower 
mean score of 19.13 (SD = 8.24) than the two experimental groups. 

The assumption of equal variance (Levene’s Test of Equality of Error Variances) was tested and groups 
were found to be homogenous (F = .18,/? = .83). Therefore, the data satisfied ANOVA assumptions. In addition, the 
assumption of normality (Shapiro- Walks Test of Normality) was met (S-W= 1.99,/? = .86). Therefore, non- 
parametric tests were not necessary. 

Table 1 

Sum mary Statistics on Comprehension Posttest Scores 



Summary 

Indications 


Collaborative Group 


Individual 

Group 


Control 

Group 


Minimum 


5.00 


4.00 


0.00 


Quartile I 


26.00 


24.00 


15.50 


Median 


30.50 


30.00 


19.00 


Quartile 3 


38.00 


35.50 


25.00 


Maximum 


49.00 


47.00 


38.00 


Mean 


31.39 


29.10 


19.13 


Standard Deviation 


8.46 


8.85 


8.24 


Sample Size 


62 


59 


40 



The one-way ANOVA results indicated that a significant difference existed among the control, individual, 
and collaborative groups on the mean scores of the comprehension posttest, F = 26.62 {p < .05.) as seen in Table 2. 

Table 2 

One Way ANOVA Summery Table for Effects of the Types of Treatment and 

Co mprehension Posttest Scores 



Source 


Sum of Square 


Df 


Mean Square 


F 


Significance 


Group 


3893.77 


2 


1946.89 


26.63 


.00* 


Error 


11552.48 


158 


73.12 







< .05. 

The Tukey Honestly Significant Difference (HSD) post hoc test was performed for group comparison. The 
Tukey HSD post hoc test revealed that the group that individually generated concept maps (individual group) 
significantly outscored the group that did not generate concept maps (control group). The group that collaboratively 
generated concepts maps (collaborative group) significantly outscored the group that did not generate concept maps 
(control group) in its performance (see Table 3). Cohen’s d indicated a positive large effect size {d =1.17) for the 
comparison of the individual concept mapping group and the control group as seen in Table 3. Additionally, 
Cohen’s d indicated a positive large effect size {d = 1.47) for the comparison of the collaborative concept mapping 
group and the control group. The effect size indicated that training students to individually or collaboratively create 
concept maps on the computer and then encouraging them to generate concept maps while studying had positive 
effects on middle school students’ science concept learning. 
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Also, students’ learning of visualization skills and the generation of concept maps on the computer during 
study time resulted in a significant positive effect on science concept learning. Therefore, the first research question 
asking if middle school students who collaboratively or individually generate computer-based concept maps perform 
better on a comprehension tests than those who do not generate computer-based concept maps was answered 
affirmatively. 

In addition, the Tukey HSD post hoc test revealed that the group that collaboratively generated concepts 
maps (collaborative group) did not significantly outscore the group that individually generated concept maps 
(individual group) in its performance. Cohen’s d only indicated a small effect size {d = 0.27) for the comparison of 
the individual concept mapping group and the collaborative concept mapping group as seen in Table 3. Although 
students who generated concept maps on the computer during study time performed significantly better than those 
who did not, there was no significant difference between the individual and the collaborative groups (Cohen’s d = 
0.27). Therefore, the second research question asking if middle school students who collaboratively generate 
computer-based concept maps perform better on a comprehension test than those who individually generate 
computer-based concept maps was answered negatively according the Tukey HSD post hoc test results. 

Table 3 

Post Hoc Test Table for the Effects of the Types of Treatment and Comprehension 
Posttest Scores 



Tukey HSD 
(I) Group 


(J) Group 


Mean Difference 


Sig 


Effect Size 


Control 


Individual 


9.98 


.00* 


Cohen’s d =1.17 
r =0.51 


Control 


Collaborative 


12.26 


.00* 


Cohen’s d =1.47 
r = 0.60 


Individual 


Collaborative 


2.29 


.31 


Cohen’s d = 0.27 
r =0.15 



< .05. 

Effects of Individually or Collaboratively Generating Concept Maps on Quality of Concept Maps 

To determine whether or not concept maps generated collaboratively exceed the quality of concept maps 
generated individually, the one-way analysis of variance (ANOVA) was applied using treatment group as the 
independent variables and rubric scores for assessing the quality of concept maps as the dependent variable. 

The assumption of equal variance (Levene’s Test of Equality of Error Variances) was tested and groups 
were found to be homogenous (F = 3.51,/? = .06). Therefore, the data satisfied ANOVA assumptions. In addition, 
the assumption of normality (Shapiro- Walks Test of Normality) was met (S-W = .98,/? = .44). Therefore, non- 
parametric tests were not necessary. 

The analysis revealed that means differed across groups (see Table 4). The mean score on the quality of 
concept maps for the collaborative group was 141.27 (SD = 30.10), while the individual group yielded a 
significantly lower score of 108.74 (SD = 40.31). 

Table 4 

Sum mary Statistics on Rubric Scores 



Summary Indications 


Collaborative Group 


Individual Group 


Minimum 


94.70 


19.70 


Quartile I 


114.05 


85.35 


Median 


143.30 


101.10 


Quartile 3 


157.05 


139.25 


Maximum 


215.80 


182.00 


Mean 


141.27 


108.74 


Standard Deviation 


30.10 


40.31 


Sample Size 


31 


59 



The one-way ANOVA results indicated that a significant difference existed between the individual and the 
collaborative groups on the mean scores of the rubric assessing the quality of the concept maps, F = 15.58 (/? < .05.), 
as seen in Table 5. The results revealed that the students who collaboratively generated concept maps significantly 
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outscored those who individually generated concepts maps in their performance on the quality of concept maps. 
Cohen’s d indicated a positive effect size {d =0.91) for the comparison of the group who collaboratively generated 
computer-based concept maps and the group who individually generated computer-based concept maps on the 
quality of concept maps. Therefore, the third research question asking if the quality of concept maps generated 
collaboratively will exceed the quality of concept maps generated individually was answered affirmatively. 

Table 5 

One-Way ANOV A Summery Table for Effects of the Types of Treatment and the 

Rub ric Scores for Assessing the Quality of Concept Maps 



Source 


Sum of Square 


Df 


Mean Square 


F 


Significance 


Group 


21496.63 


1 


21496.63 


15.58 


.00* 


Error 


121419.81 


88 


1379.77 






Effect Size (individual 


vs. collaborative): Cohen’s d = 


0.91 










r = 


0.42 







< .05. 

The results of the qualitative analyses conducted to answer the fourth research question indicated that the 
attitudes toward concept mapping for science concept learning were positive. Most students agreed that concept 
mapping was a good technique for studying. Both the collaborative group students and the individual group students 
thought that creating concept maps using the computer program was helpful and useful to study science concepts. 
Their reasons could be summarized as follows: (a) students said that concept mapping helped them organize 
information leading to better understanding and the ability to answer questions easily; (b) concept mapping assisted 
them in memorizing the science concepts and helped them retain the learned concepts to prepare for an exam; and 
(c) computer assisted concept mapping was a useful technique for adding, deleting, revising and saving concept 
maps on computers. In summary, creating concept maps using the computer program. Inspiration™^ provided 
students with a useful learning strategy and a positive learning experience. 

An interesting finding was that the students in the collaborative group were more positively engaged in 
their studying than the students in the individual group. Only 52% of the individual group students thought that 
working by themselves was helpful and useful to study science concepts while 70% of the collaborative group 
students thought that working with a partner was helpful and useful to study science concepts. According to this 
finding, the researcher inferred that students’ positive attitudes toward collaboration positively influenced the results 
on the quality of concept maps. 

The results of the qualitative analyses indicated that the control group and both experimental groups’ 
students applied different study strategies for science concept learning as follows: The control group students 
studied using their own rote learning strategies (read, highlight, underline, or memorize) while both computer-based 
concept mapping groups’ students (individual and collaborative) engaged in meaningful learning to study 
relationships between concepts and links that reflected their own understanding about conceptual science 
knowledge. This finding explains differences in comprehension between the control group and the concept mapping 
groups. 



Conclusion 

When students create concept maps with peers in collaboration, they discover, construct, and become aware 
of their own cognitive structures by representing and explaining their concepts and ideas. Collaborative learning 
facilitates divergent ways for learners to think and prompts students to consider different perspectives of a problem. 
Also, collaborative learning stimulates critical and creative thinking (Stoyanova & Kommers, 2001). 

In this study, students who collaboratively generated concept maps in pairs were actively engaged in 
creating concept maps for science concept learning, and their performance significantly exceeded students who 
individually generated concept maps on the quality of concept maps. Comparing the individual group and the 
collaborative group on the quality of concept maps, the collaborative group’s concept maps showed more valid 
propositions and relationships between concepts and links than the individual group’s concept maps. The students 
who created concept maps with pairs made more complex maps that included concepts, subconcepts, and linkages 
that constructed propositions. 
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Technology Integration and Teacher Training: The Effectiveness of 
Embedding Technology into Education Coursework 

Elaine Lawrence, Ph.D. 
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Technological literacy is quickly becoming an educational standard in the classroom. Increasingly, 
educators are expected to model technological literacy and integrate technology into the curriculum. 
Teachers who exude confidence in their technological abilities are more likely to integrate technology into 
the classroom curriculum (Jao, 2001). 

Initially, many teacher education programs require pre-service teachers to enroll in an introductory 
computer science course to obtain training in fundamental computer skills. Maeers (2000) asserts that a 
single technology course is insufficient training to generate technologically literate teachers with the 
confidence to integrate technology into the curriculum. For this reason many teacher education schools 
have been slowly altering their approaches to technology’s role within the curriculum. Teacher technology 
requirements that began as a detached introductory computer course have become either a separate 
educational technology course incorporating skills training with curriculum integration or have included 
adding technological aspects to educational methods courses and student teaching experiences (Brown, 
Appelman, Green, & Hansen, 2000). 

With this change in focus the need to study the use of technology as an instructional tool is imperative 
because of the implicit impact on future classrooms (Kjetsaa, 2002). Future educators are expected to fully 
implement technology integration into the curriculum while having had only minimal exposure to computer 
applications via introductory computer science and methods courses that do not allow for recurring practice 
in using technology to augment K-12 student learning (Beyerbach, Walsh, & Vannatta, 2001). 

Inclusion of classroom technology use training in methods classes has resulted in increases in pre- 
service teacher confidence and proficiency levels regarding integrating technology into classroom 
instructions and lessons and the learning environment (Halpin, 1999). Subsequent recommendations 
suggested the value of integrating technology training earlier within teacher education coursework 
(Beyerbach, Walsh, & Vannatta, 2001; Krueger, Hansen, & Smaldino, 2000; Ropp, 1999, Stuhlmann & 
Taylor, 1999). 

The constructivist model aligns with the fundamental belief that the knowledge base is constantly 
fluctuating and becomes stronger through discovery methods of testing and reevaluating. Since computing 
technology continually changes, teachers need to develop and maintain technology skills in order to 
maintain relevant technology integration within their classrooms. It is recommended that education 
programs need to build self-efficacy and self-confidence with computing technology integration and 
constructivist teaching (Blocher, deMontes, Tucker, & Willis, 2000). However, without courses that 
immerse undergraduate students in a constructivist approach in the many uses and applications of 
computing technology in the classroom, many of them will continue to view computing technology as an 
add-on teaching tool rather than as a device that naturally complements a constructivist approach within all 
academic subject areas. This research sought to identify specific skills and define constructivist methods 
for inclusion of technology into the pre-methods education curriculum that will prepare pre-service teachers 
to effectively infuse computing technology into their lesson plans. 

Relevance and significance 

Teacher education programs are expected to facilitate the development of teachers who are computer 
literate and able to integrate technology into the classroom (Beyerbach, Walsh, & Vannatta, 2001). 
Therefore, learning to incorporate technology into classroom learning is becoming a focal point in teacher 
education programs. Many teacher preparation programs require pre- service teachers to complete an 
introductory computer science course in an attempt to produce computer literate teachers who will 
hopefully use technology effectively within their classrooms (Abbott & Faris, 2000). But a focus on 
computer applications alone fails to prepare pre-service teachers to use technology as an effective teaching 
tool (Gibson, 2002). 
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Schmm (1999) summarizes this by stating, “It is important to look carefully at how teachers learn 
about technology, for they are quite clearly the key to transforming teaching and learning” (p. 83). Reports 
indicating that less than one-fifth of our nation’s educators feel prepared to integrate technology into the 
classroom send a clear signal that a vital element is missing in teacher preparation programs. Efforts to 
include better preparations have begun for pre-service teachers within methods coursework and during 
student teaching, but the foundation needs to be laid earlier in the education program. 

Laffey and Musser (1998) suggested that teacher education programs need to incorporate technology 
into the curriculum in order to motivate teacher candidates to creatively utilize technology. Pre-service 
teachers who experience modeling of relevant usage of technology will be more likely to implement this 
approach within their own classrooms. Recommendations from studies that infuse technology into 
education methods courses concur that technology training should begin early in the education program - 
into education courses taken prior to methods coursework - and continue throughout. Studies of this earlier 
infusion are needed in order to determine if technology integrated into pre-methods education courses have 
a positive impact on the attitudes and confidence levels of pre-service teachers. 

Globally, technology is developing exponentially and access to information is virtually available 
almost anywhere at anytime. Technology has been and continues to transform education just as it does 
businesses and within our personal lives. If current educational systems adapt appropriate technological 
functions too slowly, then they will be preparing students for a world that no longer exists. Educators need 
to model how best to incorporate technological changes and become process instructors and purveyors of 
guided learning - to use technology to “teach students how to research, what to do with the information 
gathered, and how to use it to solve problems” (McCain & Jukes, 2001, p. 1 14). 

Hypotheses investigated 

This study measured and compared the self-efficacy and confidence levels of pre-service teachers who 
had participated in an introductory computer course to those who had participated in pre-methods education 
courses with constructivist-based technology integration approaches. It also measured and compared the 
self-efficacy and confidence levels of pre-service teachers who had participated in an introductory 
computer course and a pre-methods education course with constructivist-based technology integration 
approaches to those who experienced only the introductory computer course or one of the education 
courses with the constructivist-based technology integration approaches. Baseline data from the findings 
were utilized to examine the educational technology aspects of the teacher education program. 

The purpose of this study was to compare the impact of embedding technology training into three pre- 
methods teacher education courses against the traditional approach of requiring education majors to enroll 
in only one introductory computer course. These education courses were prerequisites for enrollment into a 
methods course. The hypotheses to be investigated were: 

• Hypothesis One: Prior to methods enrollment, education majors who experience performance- 
based technology integration within core education coursework will have significantly different 
self-efficacy levels regarding technology integration than those who rely solely on the introductory 
computer course for their technological base. 

• Hypothesis Two: Prior to methods enrollment, education majors who experience performance- 
based technology integration within core education coursework will have significantly different 
confidence levels regarding technology integration than those who rely solely on the introductory 
computer course for their technological base. 

Previous studies addressed the impact of incorporating technology into education methods coursework 
and student teaching experiences for specific education majors yet these did not investigate: the impact of 
incorporating technology into pre-methods education coursework among a heterogeneous representation of 
education majors, a comparison of perceived skill attained by the pre-service teachers after taking the 
introductory computer course and/or the pre-methods courses with technology embedded within the 
coursework. Jao (2001) states that the attitude of the teacher can have an impact on the view of the students 
and Rogers (as cited in Kjetsaa, 2002) emphasizes the idea that teacher modeling of technology uses aids in 
the diffusion of knowledge in the classroom. Thus, the pre-service teacher needs to develop skills in 
technology and modeling of technology integration. This study provides insight into the impact of 
integrating technology into pre-methods education coursework - utilizing the ISTE 2000 standards - on the 
perceived skill sets of pre-service teachers. This insight can be used to develop teacher education 
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curriculum as a means of producing teachers with the skills necessary to integrate technology into the K-12 
classroom. 

Limitations 

Many colleges and universities are embedding technology integration into education methods courses 
and student teaching experiences. This study focused solely on integrating technology into three pre- 
methods courses required by every education major attending the State University of New York - College 
at Oneonta during the Fall 2003 semester: Issues in Education (EDUC 106), Diversity and Teaching 
(EDUC 201), and Philosophy and Foundations of Education (EDUC 206). Data was collected to compare 
self-efficacy and confidence levels of students receiving the constructivist-based technology integration 
treatment in selected sections of EDUC 106, EDUC 201, and EDUC 206 with those of education majors 
enrolled in Introduction to Computers (CSCI 100) during the Fall 2003 semester. Because students usually 
pre-register for these classes, the sample groups tended to be more in line with convenience sampling (Gay 
& Airasian, 2000). 

The comparison of two different curricular approaches is typical of experimental research. This study 
involved comparing specific college courses (EDUC 106, 201, and 206) that integrate technology skill 
practice and links to pedagogy to the more traditional introductory computer course that focused on six 
technology skills: spreadsheet, database, web pages, computer communications, word processing, and 
multimedia. All participants completed the same pre-survey to establish baseline data that was compared to 
data collected from the post-surveys that all participants completed. 

Participants who were Education majors and enrolled in CSCI 100 for Fall 2003 completed pre- 
surveys within the first few weeks of the semester. The data provided by Education majors enrolled in 
CSCI 100, but not in the six selected sections of the three education courses receiving the constructivist- 
based technology integration treatment, were utilized for analysis to answer Hypotheses 1 and 2. The 
surveys had items for each of the six skill areas as well as the perceived confidence and self-efficacy 
regarding technology integration. These students experienced the traditional skills approach and then 
completed post-surveys at the end of the semester. The statistical analysis indicated any change (increase, 
decrease, or no significant difference) from pre- to post-survey. 

Students enrolled in the test sections of EDUC 106, 201, and 206 selected for the study also completed 
the same pre- and post-surveys as the CSCI 100 students. The students in each of the EDUC courses had 
five of the six (spreadsheet, database, web pages, computer communications, multimedia, and word 
processing) skill areas incorporated into their education coursework. The constructivist approach involved 
the participants utilizing the technology skills as a means of completing tasks that have classroom 
applications. They reflected upon the applications to the classroom as a means of constructing their 
understanding of technology skills and integration. Prior to treatment beginning these students also 
completed the pre-surveys. These students experienced a constructivist approach and then completed post- 
surveys at the end of the semester. The statistical analysis indicated any change (increase, decrease, or no 
significant difference). The data addressed in each of EDUC courses were compared with the CSCI 100 
data for the same five areas. This indicated if there was a significant difference as specified in the 
hypotheses. 

Summary 

The quantitative and qualitative data collectively reflected positive self-efficacy and confidence levels 
for all the approaches tested. Participants experiencing the embedded constructivist approach, the standard 
introductory computer course, or those with both indicated that technology integration is possible within 
their future classrooms once these pre-service teachers become teachers. 

The literature review indicated that technology integration should begin earlier within the college 
education program. It was suggested technology integration should appear prior to methods coursework. 
The results of this study appear to align with this premise since the most significant changes occurred 
within the initial college education course (EDUC 106). If the initial education course embeds technology 
that affords pre-service teachers successful opportunities to develop perceived self-efficacy and confidence 
levels of at least four on a five-point Likert scale then successive courses can focus on developing the 
applications of technology in the classroom so that pre-service teachers entering their methods coursework 
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are able to elaborate on integration possibilities beyond their knowledge of and experience with the basic 
levels exhibited in the interviews. 

Two of the three tasks utilized in the initial education course have since been added as tasks for all 
sections of that course. Qualitative data appears to confirm the belief that pre-service teachers entering 
methods are able to develop integration activities into early lesson development. This needs to be validated 
via quantitative data collection. 
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Abstract 

This study examined, in a vicarious learning environment, the effects of group discussions and question 
prompts on pre-service teachers’ knowledge/skills and self-efficacy judgments for technology integration. Sixty-five 
students enrolled in an introductory educational technology course at a large Midwestern university participated in 
the study. Vicarious experiences for technology integration were delivered through VisionQuest, a Web site that 
provided examples of successful technology integration. A 2x2 factorial research design was implemented with the 
use of group discussions and question prompts as independent variables. Dependent variables were students’ 
perceptions of knowledge, skills, and self-efficacy for technology integration measured through pre and post 
surveys. While three of the four conditions showed significant increases in perceptions of knowledge and skills, 
ANOVA results showed no significant differences among treatments. Results suggest that vicarious experiences 
alone can be effective, and that the addition of more conditions may have distracted students from the 
content/message of the teacher models highlighted in VisionQuest. 



Introduction 

Vicarious learning occurs when learners observe the actions of others and decide which actions will be 
effective or non-effective for their own enactment of a task (McCown, Driscoll, & Roop, 1996). From a social 
cognitive perspective (Bandura, 1997; Schunk, 2000), vicarious experiences can accelerate learning over what 
would be possible if we had to perform every behavior ourselves in order to learn. This is especially true when 
trying to learn complex skills such as riding a bike, swinging a golf club, or using computers in the classroom. By 
observing experts, teachers, and other models, learners may get a head start toward their own mastery of difficult 
tasks. 

Vicarious experiences have been used to increase students’ self-efficacy, or confidence, for performing 
tasks similar to those performed by the observed models (Bandura, 1986; Ertmer, Conklin, Lewandowski, Osika, 
Selo, & Wignall, 2003; Gist, Schwoerer, & Rosen, 1989). According to Bandura (1997), self-efficacy comprises 
personal beliefs about one’s capability to perform at specified levels and, as such, is considered to be the “key factor 
of human agency” (p. 3). While not as strong of a source of efficacy information as personal mastery experiences, 
vicarious experiences often offer a more feasible method for enhancing pre-service teachers’ self-efficacy for 
technology integration, especially given the lack of resources and logistical difficulties involved in providing 
students with relevant mastery learning opportunities (Albion & Gibson, 2000; Ertmer et al., 2003). (Note: 
Additional sources of efficacy information [e.g., social persuasion and physiological states] are less powerful and 
were not considered relevant to this study.) 

Technology also offers an effective means for delivering vicarious experiences. In a study that explored the 
effectiveness of electronic models for increasing preservice teachers’ self-efficacy for technology integration 
(Ertmer et al., 2003), results showed a significant increase in students’ judgments of confidence after viewing 
successful models in a hypermedia environment. Albion and Gibson (2000) obtained similar results: After observing 
realistic examples of technology integration in an interactive multimedia environment, preservice teachers showed a 
significant increase in their self-efficacy for technology integration. Along similar lines, Driscoll and her colleagues 
(2003) demonstrated the benefits of observing dialogue-like discourse (as opposed to monologue-like discourse) that 
was modeled by a virtual tutor in a computer-supported environment. That is, overhearing dialogues containing deep 
questions helped learners recall information and thus, increased their learning outcomes. Taken together, these 
studies suggest the possibility of increasing learning and self-efficacy through the use of vicarious experiences 
delivered via computer-based environments. 
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Despite the many benefits noted in the literature, vicarious learning is not without its challenges. Since 
vicarious learning is, in and of itself, a fairly passive learning activity, learners may disengage from the activity 
entirely or fail to attend to critical aspects of the performance, thus missing out on the important benefits to be 
gained. Given these potential difficulties, it may be beneficial to combine vicarious experiences with additional 
strategies that will enable students to become, and stay, mentally involved in the experience. For example, Wang, 
Ertmer, and Newby (2004) demonstrated that vicarious experiences and goal setting, combined, were more effective 
than either strategy alone for increasing preservice teachers’ self-efficacy for technology integration. According to 
Schunk (2000), discussion groups and questioning are both effective methods for promoting active learning, and 
thus may increase students’ engagement in the vicarious learning experiences. 

Potential Benefits of Group Discussion 

According to Koschmann, Kelson, Feltovich, and Barrows (1996), meaningful group discussions can lead 
to cognitive benefits by engaging students in deep reflections on their ideas. By exchanging ideas and considering 
others’ perspectives, learners are prompted to reflect on their existing ideas as well as to integrate new ideas into 
their existing knowledge. Also, the cognitive processes involved in asking questions, providing explanations in 
response to questions, and elaborating on one’s ideas to provide these explanations, all contribute to learning 
(Cohen, 1994; Slavin, 1996). Cochran-Smith and Lytle (1999) described how teachers’ engagement in collaborative 
dialogue within a community of inquiry enabled them to share knowledge and co-create new understandings. Albion 
and Gibson (2000) emphasized the importance of including collaboration in multimedia-based, problem-solving 
environments, stating that the efficacy of problem-based learning stems from discussions among group members. 

Results from Levin’s (1995) study on the contributions of case discussions to teachers’ thinking, indicated 
that discussions helped student teachers and less experienced teachers clarify and elaborate on their ideas about 
issues in a case. Gokhale (1995) explored the effects of collaborative learning on drill-and-practice and critical 
thinking tests, and concluded that collaborative learning had a significant positive effect on critical thinking by 
helping students learn from each other’s experiences and knowledge, and by stimulating more in-depth reflection. 

Research also supports the hypothesis that group discussions can contribute to increased self-efficacy. In 
Rushton’s in-depth qualitative study (2003), findings showed that preservice teachers’ interactions with students and 
mentoring teachers helped them “cope with their doubts and abilities” (p.l67), which led to increases in self- 
efficacy. Wilson (1996) examined the effectiveness of field experiences in a teacher certification program for 
elementary science, math, and technology students and noted that participants preferred small group discussions, 
which also increased their self-efficacy. Furthermore, learners have been shown to gain confidence and improve 
their performances when they observe models who initially showed the same fears as themselves, but who gradually 
reached mastery levels of performance (Driscoll, 2000). It was hypothesized that student interactions within 
discussion groups, as used in the current study, would help learners develop self-efficacy in similar ways. That is, 
learners’ self-efficacy was expected to increase if they observed multiple people, similar to themselves, discussing 
and solving problems related to technology integration (Brown & Inouye, 1978; Schunk, 1987). 

Still, not all studies indicate that group learning is more effective than individual learning. A study by 
Snyder and Sullivan (1995) reported that when given the learning goal of changing misconceptions, students 
working individually scored significantly higher on posttest scores than those who worked in groups. Research 
specifically aimed at identifying the relationship between group discussions and learning in computer-based 
environments also demonstrated no significant differences in achievement between students working in groups and 
students working individually (Carrier & Sales, 1987; Werner & Klein, 1999). Contradictory findings on the 
effectiveness of group work in computer-based and face-to face environments, to date, suggest the need to explore 
how peer interactions affect both learning and self-efficacy outcomes when used in combination with vicarious 
experiences. 



Potential Benefits of Question Prompts 

Questioning is another strategy that might increase self-efficacy and learning, especially when used in 
combination with group discussions. For example, evidence for the relationship between questioning and self- 
efficacy can be found in a study by Powell and Ramnauth (1992), which suggested that teachers’ confidence 
increased when teachers made comments or suggestions and asked questions to stimulate the thought processes of 
each other. Additional studies suggest that by asking questions, peer interaction can be structured in ways to 
promote both cognitive development and self-efficacy. For instance, a study by Ge and Land (2003), which 
investigated the effectiveness of question prompts and peer interactions when used during tasks involving ill- 
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structured problems, revealed that question prompts facilitated students’ performance. In their study, qualitative data 
indicated that students’ thinking was enhanced during peer interactions in terms of problem representation, 
explanation, justification, monitoring, and evaluation. Van Zee and Minstrell (1997) suggested that when teachers 
ask reflective questions (i.e., questions that elicit further thinking about a topic), they encourage their students to 
clarify their meanings, consider various perspectives, and monitor their own discussions and thinking. 

Certain types of prompts have been shown to be effective in computer-based environments as well. Davis 
and Linn (2000) demonstrated that for knowledge integration to occur, self-monitoring prompts, which helped 
learners reflect on the planning and progress of activities, were more effective than activity prompts that simply 
directed learners’ attention to completing the activities. A study by Han, Crooks, and Xie (2005) showed that using 
unstructured question prompts (prompts which appeared separately from, but not inserted within, the text passage on 
the computer screen) can be effective for performance transfer especially when learners have to key in answers to 
the computer program. 

It should be noted, however, that research findings on the effectiveness of questioning have been 
inconsistent. Wang (2001) explored the effects of summarization and structured questions, hypothesizing that 
structured questions would activate metacognitive and critical thinking skills. However, results of this study 
indicated no statistical effects of either treatment on transfer tasks. 

While there are inconsistent reports on the effectiveness of group work and question prompts in both face- 
to-face and computer-based environments, in general, the literature suggests that both methods may have a positive 
effect on learning by facilitating reflection and critical thinking. This suggests that group discussions and question 
prompts may enhance the use of vicarious experiences with preservice teachers and thus may have an impact on the 
growth of their knowledge, skills, and self-efficacy. 

The current study was designed to examine, in a vicarious learning context, the impact of group discussions 
and question prompts on students’ knowledge/skills and self-efficacy judgments for technology integration. 
Specifically, this study was guided by the overarching research question: What are the effects of group discussions 
and question prompts on preservice teachers’ judgments of computer competency and computer self-efficacy in a 
vicarious learning environment? 

Based on the literature above, it was hypothesized that students who participated in group discussions while 
observing exemplary technology-using teachers (i.e., participated in vicarious experiences) would have increased 
perceptions of competency and self-efficacy in this area. It was also hypothesized that students who received 
question prompts to guide their reflections during vicarious learning experiences would demonstrate a significant 
increase in judgments of competency and self-efficacy for technology integration. Finally, it was hypothesized that 
students who participated in group discussions and received question prompts would have the greatest increase in 
judgments of competency and self-efficacy compared to students who received either one of these conditions alone. 

Methods 



Research Design 

A 2 X 2 (group discussion X question prompts) research design was used to examine the impact of group 
discussions and question prompts on preservice teachers’ judgments of competency and self-efficacy for technology 
integration. The independent variables were combined to form four experimental conditions: (1) NGD/NQP (the 
control group, that is, no group discussion and no question prompts), (2) NGD/QP (no group discussion, but with 
question prompts), (3) GD/NQP (group discussions, but no question prompts), and (4) GD/QP (group discussions 
and question prompts). The dependent variables comprised participants’ pre- and post-perceptions of their 
competencies (knowledge and skills) and self-efficacy for technology integration. 

Participants and Sampling Method 

Participants were solicited from 420 students enrolled in an introductory educational technology course 
during the spring of 2005 at a large Mid-western University. The 2-credit course consisted of weekly 1-hour lectures 
and 2-hour lab sessions, and was designed to provide students with the foundational knowledge and skills necessary 
to integrate technology into K-12 classrooms. Participation was solicited during the 8th week of the semester during 
18 lab sessions that were taught by instructors unrelated to this study. An approved human subjects protocol was 
used and 65 students participated in the study. 



218 




Participants included 43 females and 22 males, ranging in age from 18 to 41 years (M = 19). The majority 
were freshmen (n=36) and sophomores (n=21). Participants were majoring, primarily, in either secondary (n=36) or 
elementary (n=18) education. At the time of the study, 92% of the students indicated that using computers was 
somewhat or very easy. In addition, 86% of the participants noted that it was important, or very important, to use 
computers in the K-12 classroom. Results of initial ANOVAs indicated no significant differences among the 
treatment groups in terms of age, class, major, or pretest scores on either the competencies or self-efficacy subscales. 

Instrument 

The Technology Integration Knowledge, Skill, and Self-efficacy Assessment (TIKSSA) survey, used to 
measure the dependent variables, was developed after reviewing similar surveys in the literature (Schunk & Ertmer, 
1999; Wang, Ertmer, & Newby, 2004). TIKSSA was examined for both content and construct validity. A review by 
a content expert in the area of self-efficacy provided evidence of content validity. Evidence of construct validity was 
gathered by conducting an exploratory factor analysis (EFA). EFA was chosen based on the fact that TIKSSA was a 
new scale, and it was necessary to identify factor patterns for each subscale. For EFA, principal factors analysis 
methods and the scree test were used to extract factors and determine the number of factors. Results indicated that 
for both subscales items loaded heavily on one factor, respectively: Skills/knowledge scale with an eigenvalue=8.52, 
explaining .78 of the total variances, and the self-efficacy scale with eigenvalue=9.69, explaining .85 of total 
variance. While readers should not consider the results from this EFA to be entirely reliable, due to the small sample 
size, it does provide preliminary evidence of construct validity. 

To complete the TIKSSA, participants rated their levels of agreement (from 1-not well to 4- very well) on 
32 Likert-style items reflecting their current perceptions of competencies (knowledge and skills) and self-efficacy 
for tasks such as designing and managing technology integrated classrooms and overcoming barriers. Sixteen items 
were used to measure students’ perceptions of knowledge and skills (e.g., ‘T know how to discuss my vision of 
technology integration with school colleagues or supervisors.”); an additional 16 items were used to measure self- 
efficacy (e.g., ‘T feel confident that I can implement my vision of technology integration within my future 
classroom.”). Surveys were accessed and completed online: the pre-survey was completed approximately a week 
before the study; the post-survey was completed immediately after participation in the treatment. Cronbach alpha 
coefficients indicated that the two subscales were highly reliable (knowledge/skills = .95; self-efficacy = .96) 

Instructional Materials 

Students in all four treatments reviewed VisionQuest (Ertmer et al., 2003) before participating in the 
treatment conditions. VisionQuest is an instructional website that allows users to explore how classroom teachers 
effectively integrate technology into their classrooms despite differences in subjects, resources, and student 
characteristics. 

The VQ website consists of three main sections (planning, implementation, and assessment) related to 
teachers’ efforts toward achieving technology integration. Teacher cases are presented through various media 
including video, text, and graphics; relevant artifacts (lesson plans, assessment tools, student work, etc.) are also 
included. 

Procedures 

The entire experiment was conducted over four days during the ninth week of the semester. The researchers 
assigned treatment conditions based on the number of students who had signed up for each time slot and the overall 
numbers for each experimental group. Students learned which treatment they would receive after they arrived at the 
labs. Due to scheduling conflicts, a small number of participants (n=7) completed the study two weeks later. 

As participants arrived at the lab, they received two handouts: 1) VisionQuest User’s Guide, and 2) General 
or Question Prompts Worksheet. The first handout familiarized students with the website and outlined the 
procedures for the study. The second handout guided students’ individual reflections or group discussions over the 
contents of VisionQuest. Depending on their treatment conditions, students received either a Question Prompt 
Worksheet (handout containing question prompts and space to record notes) or a General Worksheet (checklist of 
items to view, with space to take notes). Table 1 summarizes the types of worksheets and activities given to each 
experimental group. More specific descriptions of the worksheets are provided below. 
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Table 1. Treatment Conditions in 2 x2 Factorial Design 



Question Prompt / 
Discussion Group 


No Question Prompt (NQP) 


Question Prompt (QP) 


No Discussion 


General worksheet 


Question prompt worksheet 


Group (NDG) 


Individual work 


Individual work 


Discussion Group 


General worksheet 


Question prompt worksheet 


(DG) 


Group discussion 


Group discussion 



Question Prompts Worksheets (QP) 

Participants in the NGD/QP and GD/QP groups received a set of 1 1 questions to guide them as they 
explored the VisionQuest website. Specific prompts included, for example: “Which class activity, from among all 
three cases, did you think was most innovative and effective? Why?” and “How were teachers’ visions reflected in 
their students’ learning?” The questions were designed to stimulate students’ critical thinking on issues related to 
technology integration. Researchers emphasized the purpose of these questions on the worksheet and also orally at 
the start of the lab sessions. 

General Worksheet (NQP - No question prompts) 

Participants in the groups without question prompts were given checklists designed to ensure that they 
explored important components of VisionQuest. The difference between this worksheet and the question prompts 
worksheets was that it asked the students to take notes on general ideas related to each of the components of 
VisionQuest rather than responding to specific questions or prompts. 

Discussion Groups (DG) 

Students in the DG/NQP and DG/QP groups participated in discussions after exploring components of 
VisionQuest independently. Most groups had three participants. For each time slot designated for group discussions, 
if the number of students was not adequate to make groups of three, groups of four were organized by the 
researcher. Participants were randomly assigned to groups as they arrived at the lab. Either the general worksheet or 
the question prompts worksheet was provided to each group. Each group was required to submit one worksheet to 
the researcher. The total time spent reviewing and discussing VisionQuest was also recorded. Discussions lasted 
from 60-90 minutes, depending on the interaction level of the participants. Participants who were not assigned to the 
group discussion treatments (NGD/QP and NGD/NQP) reviewed and filled out their worksheets individually. 

Results 

This study was designed to determine the effect of question prompts and group discussion on students’ 
judgments of technology integration competencies and self-efficacy while engaged in vicarious learning 
experiences. A two-way ANOVA indicated no significant effects of either independent variable (question prompts, 
discussion groups) on the dependent variables (F = 1.04, p = .38 for competency judgments; F=.57, p=.64 for self- 
efficacy). In addition, there were no statistically significant interactions between the independent variables on 
competencies (F=.l 1, p = .74) or self-efficacy (F = .00, p = .96). The calculated effect size (omega square) was .03, 
suggesting that the systematic variance was only minimally explained by the treatment condition in which the 
students participated. 

Results of paired t-tests indicated that students in all treatment groups, with the exception of the NDG/QP 
group, showed significant increases in their perceptions of knowledge and skills from pre- to post-survey. Results 
from paired t-tests also showed significant increases in judgments of self-efficacy from pre to postsurvey for 
students in the Control Group (p = .001), and the DG/NQP (p = .001). Students in both treatment groups that used 
question prompts (NDG/QP and GD/QP) did not show significant increases in self-efficacy. In addition, the students 
who used question prompts individually (NDG/QP) did not show significant increases in perceptions of knowledge 
and skills. Surprisingly, this group also had the lowest scores on the postsurvey. Means for both subscales on the 
pre- and post-surveys, as well as t and p values, are included in Table 2. 
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Table 2. Treatment Group Scores on the TIKKSA Pre- and Post-Survey 



Group 


n 




Knowledge/Skills 






Self Efficacy 








Pre- 

Mean 


Post- 

Mean 


t 


P 


Pre- 

Mean 


Post- 

Mean 


t 


P 


NDG/NQP 
(Initial Control Gp: 
Open-ended Wksht) 


14 


2.56 


3.06 


4.37 


.0008 


2.77 


3.10 


4.10 


.0013 


NDG/QP 


16 


2.49 


2.76 


1.80 


.092 


2.66 


2.88 


1.81 


.090 


DG/NQP 


18 


2.49 


3.10 


4.29 


.0006 


2.58 


3.16 


3.99 


.001 


DG/QP 


17 


2.47 


2.93 


3.88 


.0015 


2.75 


2.99 


1.11 


.2845 


NDG/NWK* 
(Control: No Wksht) 


15 


2.45 


2.90 


4.48 


.0006 


2.60 


2.97 


3.01 


.010 



{^Note. This condition was added to address initial findings suggesting that the original control group may 
have benefited from the use of an open-ended worksheet.) 

Because this result was unexpected, we analyzed the responses that students wrote on their group and 
individual worksheets to obtain further insight into how the prompts may have guided their interactions with 
VisionQuest. In general, responses of students in the question prompt groups were composed of sentences that 
tended to repeat pieces of information found on the VQ website, contrary to the expectation that they would be more 
substantive and reflective (e.g., “The teacher . . . used technology by having her students create their own songs” 
(DG/QP); . school teachers integrated their classroom to a more hands-on experience. . . . The class layout 
contained 16 computers.”). In contrast, responses of students in the non-question prompt groups appeared to be in 
the students’ own words, and tended to refer to broader ideas rather than specific facts (e.g., “Incentives are based on 
the teacher’s beliefs, and student’s motivation,” and “computers can help students judge information and make 
decisions with their own conclusions.”). 

Again, this illustrates an unexpected trend. That is, students who used a worksheet that simply asked them 
to “jot down their ideas” related to components of VisionQuest, seemed to respond in more reflective and thoughtful 
ways than those who used a worksheet that asked them to respond to specific questions (e.g., “Describe how 
teachers’ visions influenced the design of their classroom activities or materials. Give at least one specific 
example.”). An unintended effect may have occurred in which students’ attention was directed toward answering 
questions, as opposed to reflecting on what they were observing in VisionQuest. Thus, the more general worksheet 
may have actually served as the type of reflective prompt we were hoping to create with the more directed questions. 
In effect, this may have modified the condition that we intended to serve as a control into one that utilized a 
different, and perhaps more effective, type of question prompt. 

Due to these findings, an additional condition was added to the design of the study and implemented with a 
new group of 13 students enrolled in the same course the following semester. ANOVA results indicated no 
significant differences between the new and initial participants in terms of class, age, or computer skills. In the 
hopes of creating a “true” control condition, that is, one in which no prompts were provided, students in the new 
condition did not use any worksheets while exploring VisionQuest. Rather, students were simply asked to mark, on a 
provided checklist, those parts of VisionQuest they had viewed. Thus, the students in this condition did not write 
down any notes or comments as they went through the program. While results of a t-test showed a significant 
increase from pre to postsurvey on both the knowledge/skills subscale (p=.006) and the self-efficacy subscale (p = 
.01), one way- ANOVA results indicated no significant differences between this control group and the other NDG 
treatment conditions on either the knowledge (F=1.21, p = .31) or self-efficacy (F = .66, p = .52) subscales. Students 
in the initial Control Group showed the greatest increases on both the knowledge scale (t = 4.37;p = .001) and the 
self-efficacy scale (t = 4.10; p = .001). 

Discussion 



This study was designed to determine if the effects of vicarious experiences could be increased when used 
in conjunction with additional instructional strategies, known to be effective in promoting learning. For example, in 
a previous study, Wang et al. (2004) found that the use of goal setting strategies, when paired with vicarious 
experiences, significantly increased students’ judgments of self-efficacy for technology integration. In this study, we 
examined the specific benefits of pairing vicarious experiences with group discussion and question prompts. Despite 
the fact that students in all but one treatment group significantly increased their perceptions of knowledge and skills. 
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the additional use of discussions and question prompts did not create any significant differences among students’ 
perceptions across groups. In fact, students who viewed VisionQuest individually, using only an open-ended 
worksheet, appeared to make the most gains in perceptions of knowledge, skills, and self-efficacy. Adding group 
discussions and/or question prompts did not appear to enhance the benefits of these vicarious learning experiences 
over those which occurred without them. 

In this study, students who participated in group discussions while using VisionQuest did not benefit 
significantly more or less from the vicarious learning experiences than those who did not participate in group 
discussions. While previous research (Gokhale, 1995; Levin, 1995) has suggested that group discussions can 
stimulate students’ reflection and understanding of complex issues, and may raise judgments of self-efficacy 
(Wilson, 1996), this was not the case in our study. This may have been due to a number of reasons including having 
a relatively small number of participants in each treatment group, as well as a relatively limited amount of time to 
carry out a number of in-depth discussions on a variety of topics. In addition, students may not have been 
comfortable discussing their ideas with others whom they did not know very well, or simply may not have been 
motivated to do any more than was required to complete the assigned tasks. Because these activities were not part of 
their required curriculum, students may have judged that they were not particularly interesting or relevant to them. 
This may have caused them to simply “go through the motions” of discussing the content, but with no real incentive 
to reflect deeply on the questions at hand. Furthermore, adding supplementary materials or tools for group work may 
actually have had a negative effect on learning. A study by Van Boxtel, Van der Linden, and Kanselaar (2000) 
demonstrated that additional materials, such as the use of textbooks, can inhibit elaborative and constructive 
interaction among peers. Other researchers (Webb, 1989; Werner & Klein, 1999) have reported similar findings: 
group learning does not always lead to significantly greater learning outcomes compared to individual learning. 

Particularly interesting in this study were the results related to the use of question prompts. While it was 
expected, based on the literature, that question prompts would support students’ learning by stimulating deeper 
thinking about the content (Ge & Land, 2003; Van Zee & Mistrell, 1997), students in this study who did not use 
question prompts scored consistently higher on the post-survey scales than those who did. Although these 
differences between groups were not statistically significant, as demonstrated by the ANOVA results, they do 
suggest an interesting trend. 

By asking students to respond to questions that guided their learning in a specific direction, researchers 
may have inadvertently focused students’ attention on specific answers, rather than on the rich vicarious experiences 
being presented on VisionQuest. These prompts, or questions, then, may have actually distracted students from fully 
attending to the vicarious experiences they were expected to observe, thus leading to smaller gains in perceived 
knowledge, skills, and self-efficacy. While this result was not expected, there is some evidence in the literature to 
support this finding. For example, Davis (2003) found that students actually incorporated more ideas, evidence, and 
scientific principles in their reflections when guided with “generic” prompts (e.g., “Right now, we’re thinking. . .”) 
rather than more “directed” prompts (e.g., “To do a good job on this project, we need to. . .”). She speculated that 
students who received the directed prompts may not have been able to thoroughly interpret what was needed or they 
may have been superficially completing the task. This seems to have been true in our study as well. Participants who 
did not use question prompts may have had more opportunity to engage in broader reflections over the content, 
guided by their own judgment, and thus engaged in deeper reflections compared to those using more specific 
question prompts. 

While it is possible that the students in this study were not motivated or prepared to think deeply about the 
content presented (Wang, 2001), another possibility is that the relatively long list of question prompts (n= 11) may 
have caused fatigue in participants. According to Kobayashi (2005), cognitive fatigue can have a negative influence 
on note-taking during long presentations. A full review of VisionQuest, without the additional requirement of 
recording ideas gained from the site, would take 90 minutes or more to complete. Adding a relatively lengthy 
worksheet to the review requirements may have caused a loss in concentration and motivation. In contrast, the no- 
question prompt groups were asked to record ideas related to only 5 key items, a substantially smaller number. 

The results of this study suggest that vicarious experiences alone may increase learning and self-efficacy 
without the help of other methods. The addition of more conditions may have distracted students from the 
content/message of the teacher models highlighted on VisionQuest. It is possible that the cognitive load imposed on 
students during vicarious learning activities does not allow enough room for processing other, seemingly competing, 
instructional tasks. Conversely, it is possible that the extra tasks required too much additional attention and thus 
were not helpful in guiding students’ efforts on the initial task (Chandler & Sweller, 1991; Sweller & Chandler, 
1994). In addition, cautious administration of additional strategies should be considered when providing vicarious 
learning experiences. For group discussions to facilitate reflection, methods for developing group dynamics and 
increasing students’ motivation to actively participate could be necessary. Question prompts that direct reflection 
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could be more effective when they are not focused narrowly in specific directions and allow freedom for learners to 
choose their own approaches to processing the information gained from vicarious experiences. 

Limitations 

The results of this study are limited by the 1) small number of participants, 2) lack of time for group 
dynamics to develop, 3) voluntary nature of participation in study, which was to “earn” credit (students were just 
“completing a task” and may not have understood the relevancy of VisionQuest to their future career as teachers), 
and 4) structured questions that unintentionally narrowed the reflective process of learners. Assuming that group 
discussions and question prompts have the potential to increase learning outcomes, future research should 
incorporate strategies that address these limitations. For example, using VisionQuest as a required activity in a 
course might eliminate a few of the concerns noted above. That is, asking students to discuss VisionQuest, with 
others whom they know, to address a specific course goal, might have allowed the benefits of both group 
discussions and question prompts to emerge. These are fruitful areas for future work. 
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Investigating Ethical Issues Experienced by Professional Technologists in 
Online Course Design and Web-based Training Situations 

Hong Lin, Ph.D. 

Oklahoma State University 

Abstract To correspond to the AECT Code of Professional Ethics and the professional journal TechTrends' ethics 
columns, this paper provides empirical data regarding ethical issues associated with the use of instructional 
technology in design and training situations. In-depth interviews of 20 practitioners were conducted. The top three 
concerns reported were related to copyright, learner privacy, and accessibility issues. Diversity, conflicts of interest, 
and professionalism concerns also were raised. Recommendations to researchers, managers, and practitioners were 
discussed. 

Keywords: Ethics, instructional technology, training 

Course design and training is undergoing great changes due to the advancement of technology and such new 
styles of learning as online learning. For this reason, educational institutions have motivated to increasingly 
incorporate instructional technology into course design and training. Such a change, in turn, has enabled learning 
and training to become increasingly digital, mobile, and virtual. Meanwhile, the instructional technology that has 
profoundly impacted course design and training practices has also given rise to the proliferation of ethical issues 
relating to the applications of the technology. Such ethical issues include, but are not limited to, digital copyright 
infringement (Mabry & O’Driscoll, 2003), the violation of private online information (Mason, 1995), and other 
inappropriate use of instructional technology. 

To address the emerging ethical issues in the field of educational communications and technology, the 
Association for Educational Communication Technology (AECT) has established a Code of Professional Ethics for 
its members with complex and comprehensive guiding principles that aim at a wide range of professional behaviors. 
Correspondingly, the AECT professional journal TechTrends has published a series of ethics columns to interpret 
the Code established by AECT. The AECT Code of Professional Ethics and the TechTrends series have undoubtedly 
heightened the field’s awareness of the importance of ethical behaviors. 

However, according to Mabry and O’Driscoll (2003), these largely speculative discussions of ethical 
challenges and ways to navigate the challenges in these related fields are not “grounded in empirical research,” and 
“by far the least researched perspective is an empirical one that examines ethics and ethical dilemmas actually 
encountered in practice” (p. 82). Yeaman (2004) echoed this view by claiming that professional ethics and the 
misuse of technology by professional technologists “may be a rather neglected topic for research” (p. 16). 

The lack of literature in the area of ethics and instructional technology is incongruent with the fact that more 
and more practitioners are employing instructional technology on a daily basis in design and training situations. 
Actually, practitioners and researchers in the field have had a long history of examining ethical obligations and 
developing expectations for professional behavior as well as a code of ethics. The purpose of this study, therefore, 
was to identify ethical issues and coping strategies experienced by professional technologists in design and training 
situations. It was hoped that the results would contribute to the current literature about ethical obligations in the field 
of educational communications and technology. 

Problem Statement 

Little is known about the ethical challenges faced by professional technologists in online course design and 
Web-based training situations. Instead, studies associated with the use of instructional technology primarily focus on 
the implementation of technology (Bowen, 1998); the effects of technology on process variables, such as learner 
control, feedback, and usability (Torraco, 1999); principles of effective online courses (Speck, Knowlton, & Weiss, 
2000), and competences for online teaching (Goodyear, Salmon, Spector, Steeples, & Tichner, 2001). Moreover, 
studies related to application of technology are often reported in the Information Technology (IT) field rather than in 
the educational communications and technology arena. 
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Literature Review 



A review of the literature focused on ethical issues in the applications of instructional technology will provide a 
framework that will help us compare what we already know to areas identified in this study that need further 
exploration. 

Copyright 

In the literature, discussions of digital copyright mainly focus on its legal perspectives. Two legislations are 
relevant to the use of digital copyright in educational institutions. The Digital Millennium Copyright Act of 1998 
limits the liability of nonprofit institutions of higher education — ^when they serve as online service providers and 
under certain circumstances — for copyright infringement by faculty members or graduate students (UCLA Online 
Institute, 2004). The TEACH Act, an acronym for the Technology, Education, and Copyright Harmonization Act, 
gives more freedom to accredited, nonprofit educational institutions to use copyright protected materials in distance 
education — including on Web sites and by other digital means (the American Library Association web site). 

On the other hand, the literature also covers discussions of copyright from its ethical perspectives. Mabry and 
O’Driscoll (2003) pointed out that to lower the cost of designing, developing, and delivering learning content, some 
organizations reuse and reorganize learning content that is already available. In this sense, professional technologists 
are susceptible to such ethical breaches as taking others’ credit consciously or inadequately and even becoming 
involved in high-tech intellectual property crime (Mabry & O’ Driscoll, 2003). Similarly, Kebbati (2001) identified 
some unethical computer behaviors in a high school classroom such as the confusion of fair use guidelines and 
copyright clearance for teaching. In university settings, Lan and Dagley (1999) cautioned that as university 
professors busily create online courses and distribute instructional materials online, educational technologists, legal 
counsel, and university administrators need to advise faculty of the dangers of trying to do things the old way in a 
new format, that is, putting hardcopy materials online without copyright clearance. 

Confidentiality 

With its swift expansion and growing popularity, information technology provides broad access to confidential 
resources (Pourciau, 1999). Violation of individual privacy and the abuse of confidential information have steadily 
proliferated. In 2000 alone, more than 200 bills on privacy were introduced in the 106^^ Congress of the United 
States (Hatcher, 2002). These measures aim at protecting individual rights and controls of personal data. 

In design and training situations, instructional technology exacerbates the recognized confidentiality issues, 
some of which pertain to professional ethical conduct. For example, adult learners in training situations may feel 
pressured that their performance or level of competences would be divulged to their managers. The pressure is not 
imaginary as new types of technological software, such as expertise locators, can track every step of the use of 
online resources, assume a learner’s interests and potential expertise based on his/her online activities, and 
categorize the individual within the organization (Mabry & O’Driscoll, 2003). Frequently, all this occurs without the 
acknowledgement of the learners. 

Web Accessibility 

A 1997 U.S. Census Bureau report indicated that 19.6% of the U.S. population has some sort of disability such 
as visual or hearing impairments (French & Valdes, 2002). It is clear that providing equal learning opportunities to 
people with disabilities is imperative. In 1996 the Justice Department stressed that the entities under the ADA are 
required to provide effective communication through print media, audio media, or such electronic media as the 
Internet (French & Valdes, 2002). In 2001 the accessibility requirements for U.S. electronic and information 
technology took effect under Section 508 of the Federal Rehabilitation Act, which is considered to be the ADA of 
cyberspace (Waddell & Thomason, 1998). 

Other than laws that require Web accessibility, many agencies, from an ethical standpoint, have been 
contributing to the growth of Web accessibility to people with disabilities. For example, the World Wide Web 
Consortium (W3C) has earned international recognition for establishing guidelines and providing resources to the 
universal design of the World Wide Web. Web Accessibility in Mind (WebAIM) aims at improving accessibility to 
online learning opportunities for all people and, especially, helping people with disabilities who experience 
difficulty in getting access to post-secondary online learning opportunities. 

These joint efforts have been appraised by researchers. Foley and Regan (2002) stressed that “accessibility is 
the right thing to do” (p. 66) because it gives people with a disability a broader range of employment and 
educational opportunities via innovative technology. French and Valdes (2002) echoed that “the sooner we create 
and implement sustainable solutions, the sooner ALL students can participate in their right to experience the power 
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of the Internet for lifelong learning” (p. 13). However, as Camevale (1999) indicated, often those Web site creators 
of distance-education programs in higher education do not ignore the Web accessibility issue on purpose; rather, 
they don’t always realize the importance of Web accessibility. 

Diversity and Inequality 

One challenge to our increasingly diverse society is the balance between those who have power and resources 
and those who have not. As Han (1994) contends, power gaps and inequality in society are “partly attributed to 
general apathy concerning ethics and disparity in sharing knowledge discovery between ordinary people and those 
holding power” (p. 1). Instructional technology, as a powerful tool to enhance knowledge acquisition and transform 
the global information infrastructure, can be used to achieve the goals of closing the power gaps as well as widening 
the gaps. Ideally, if technology-based learning is accessible to everyone, the resulting knowledge discovery is 
helpful in removing societal inequities. On the contrary, if technology-based learning is manipulated by someone 
who uses the power inappropriately, power gaps can be widened (Han, 1994). 

Unfortunately, technology-based learning has seen limited use in “teaching diversity or developing societies 
based on equality” (Hatcher, 2002, p. 146). In other words, while technology empowers some to advance and reach 
potentials, it also reinforces traditional barriers to others. Professional technologists, as Hatcher (2002) suggests, 
should “be diligent in providing equal access to training and other technology-based, career-enhancing activities” (p. 
145). 

Technology-Based Learning 

Technology-based learning seeks to “improve learning through the metaphor of ‘instructional technologies’” 
(Hatcher, 2002, p. 146). However, certain ethical issues arise when technology-based learning is conducted. The 
first ethical issue is that available technology often overshadows learning needs or each new technology is used as a 
replacement for all existing learning methodologies (Piskurich & Sanders, 1998). In this sense, knowingly 
recommending or adopting unnecessary technologies is known as “technolust” (Bassi, Buchanan, & Cheney, 1997). 
Arguably, researchers suggest that an appropriate performance analysis, which emphasizes the analysis of 
performance gaps, learning needs, goals, and identification of the underlying causes of the problems, should be 
conducted to justify which technologies are the best fit and can supplement the intervention (Rossett, 1992; Rossett 
& Arwady, 1987). 

The second ethical challenge for professional technologists relates to the awareness of the impact of their work 
on other people. The decisions to use or dispose of instructional technologies do not stand alone with the decision 
makers. Rather, as Mason (1995) points out, any changes in “hardware, software, information content, information 
flow, knowledge-based jobs, and the rules and regulations affecting information are among the many things agents 
do that affect others” (p. 55). For this reason, professional technologists should take into consideration their actions 
and take responsibility for the impact of their actions on other people. 

Laws 

Two laws in particular address copyright issues relating to the application of instructional technology. The first 
law is The Digital Millennium Copyright Act of 1998. This law outlines copyright issues and policy goals to protect 
intellectual property and the national information infrastructure. One of the most notable messages from the law is 
that it limits Internet service providers from copyright infringement liability for simply transmitting information 
over the Internet. (UCLA online Institute, 2004). The second important law is The TEACH Act, an acronym for 
Technology, Education and Copyright Harmonization Act. It was signed into law in 2002. The TEACH Act, as 
described by the American Library Association, was long anticipated by educators and librarians in that it “redefines 
the terms and conditions on which accredited, nonprofit educational institutions throughout the U.S. may use 
copyright protected materials in distance education — including on websites and by other digital means — without 
permission from the copyright owner and without payment of royalties” (the American Library Association web 
site). 

The issues addressed by these two laws reflect a growing concern that has arisen as technological 
improvements have facilitated the transfer and use of information. The research question reflects this concern. 

Research Question 

What ethical issues associated with instructional technology do professional technologists report as having 

experienced in online course design and Web-based training situations? 
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Methodology 



Subjects 

Purposive sampling was used to recruit participants from three departments in a major Mid- Atlantic research 
university. Several criteria were used to recruit research participants from these units. The participants were required 
to have a minimum of two years of experience working with learning technologies in one of the following 
responsibilities: 1) designing and developing online courses, 2) providing instructional technology support to 
establish e-leaming environment, and 3) incorporating learning technologies into training programs. The selection 
criteria ensured that the research participants had appropriate levels of experience in utilizing learning technologies 
from which to draw upon to discuss experiences in ethical situations. 

After the directors of the three departments sent out an e-mail announcement, attached with my invitation letter, 
to their units’ list serves to encourage their staff to contact me on a voluntary basis, a total of 20 professional 
technologists were recruited in this study. Their job titles primarily include instructional designers, instructional 
technologists, and technology trainers. Eighty percent of the participants were in the age group from 25 to 45 years; 
their experience working in learning technology ranged from 2 to 15 years. Ten participants were male, ten were 
female; twelve had an Adult Education/Instructional Systems/Training background. 

Procedure 

In-depth, face-to-face interviews were used to collect data. The advantage of in-depth interviews, as Van 
Manen (1990) indicated, is that a researcher, by conducting the interview, and reading and rereading the interview 
texts, can discover, literally “hear,” the meaning of the lived experience from the experiencer’s perspectives. After a 
pilot interview, the original interview guide was revised to improve the clarity of the questions. In its final form, the 
interview protocol consisted of seven open-ended questions with suggested probes and clarifications under each 
question. These open-ended questions asked participants to recall ethical issues associated directly with their work 
activities. The length of the interviews varied with an average session lasting about 40 minutes. 

During the interviews, the interview guide was followed using a protocol that allowed for flexibility in the use 
of probes and clarifying questions. Since recalling ethical situations and being audio recorded might be stressful to 
some participants, all were assured throughout the process of the confidentiality of their responses. The interviews 
were conducted over a four-month period. 

Data Analysis 

All interviews were transcribed verbatim. Afterwards, data analysis for this study followed the constant 
comparison techniques that were described by Glaser and Strauss (1967) and Strauss and Corbin (1990). Two 
researchers (one author and one other person who was experienced in qualitative analysis but not involved in the 
study) independently read and coded the transcripts. Data analysis involved three phases. During the first phase, 
free and open-coding was used (Strauss & Corbin, 1990). This process begins by reading through the transcripts 
several times, immersing oneself in the data, underlining key words and phrases in the transcripts and making notes 
on each section of each transcription. These key words, phrases, and notes are the units of analysis that provide the 
groundwork for further analysis. 

The second phase of the data analysis process engaged axial coding. This method uses a series of procedures 
that make connections within and between groupings and allows for new combinations of data (Strauss & Corbin, 
1990). Using the condensed list that was developed in Phase 1, coders sought common and related concepts from the 
collection of instances documented in each grouping. Following the procedures of axial coding, we continued 
making connections and combining data within each grouping. During this process, some units of analysis were 
regrouped wherever necessary. Next, in Phase 3, coders examined and summarized core variables as themes and 
placed situations under appropriate themes. Using this approach, themes were generated and summarized for each 
research question. 

Validity, Reliability and Objectivity 

A synthesis of each interview was prepared and sent to the participant for content verification. All the 
participants replied and indicated that the syntheses had captured the essence of their interviews. Two people 
independently coded the transcripts and discussed and resolved differences. An audit trial — detailed field notes, 
memos, and notes— was maintained throughout the process. Two people not involved in the initial coding reviewed 
the analysis. 
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Results 



The issues mentioned most frequently by the participants are found in Table 1. 
Table 1. Summary of Ethical Issues Experienced by Professional Technologists 



Issues 


Number of Participants (n=20) 


Percentage 


Copyright 


15 


75% 


Feamer Privacy 


13 


65% 


Accessibility 


II 


55% 


Diversity 


7 


35% 


Conflicts of Interest 


4 


20% 


Professionalism/Confidence 


3 


15% 



Out of 20 participants, fifteen mentioned copyright as a major ethical issue they had to handle in their daily 
work activities. Thirteen indicated that learner privacy is an ethical issue for them, and over half indicated 
accessibility as a major concern. For the first three issues, I give verbatim examples from the transcripts. For the last 
three, due to space constraints, I give only summaries. 

Issue 1: Copyright 

The most frequent ethical issue that learning technology practitioners have experienced is digital copyright. 
There are three situations in which learning technology practitioners encountered the copyright issue. The first 
situation was usually when practitioners, especially instructional designers and instructional technology specialists, 
interacted with faculty members and needed to communicate extensively with them about copyright clearance in 
order to design online materials. One study participant indicated: 

Copyright is a big issue in our day-to-day working. We also have requests from faculty members to use things 
without getting copyright clearance, like they might give us an image or they may want to be able to use a 
piece of text from the textbook. Explaining that we are restricted in what we can do is often a big topic of 
conversation (Participant 5). 

The second situation regarding copyright was getting permission for an item to use in the work they were 
developing. This item could be a piece of text, a graphic, a Hollywood movie clip, or a music clip. In order to be 
able to use these items, practitioners had to obtain copyright clearance. One participant said: 

Our unit is heavily involved with making sure that anything we work with a project has copyright clearance, 
and that we have the right to use. We do a lot of development in house of things of our own. If we can’t get 
copyright for something, we create things for ourselves to make sure that there are no copyright issues. We 
have that in all areas (Participant 4). 

The third copyright situation was struggling to balance copyright and fair use. Specifically, a majority of 15 
participants who mentioned copyright as an ethical issue also indicated that there was a gray area between copyright 
and fair use guidelines in their job. For example, a trainer who usually prepared seminar handouts and distributed 
them in free technology seminars in the university said: 

With training, I think the biggest ethical dilemma is whether information is copyrighted or not. For example, if 
you are teaching an Intro class for Microsoft, and a lot of time because of time, instructors including myself, 
we end up just copying and pasting the how-to-step-by-step from the HEFP menus or from other materials 
[...]. Is it OK to do this? I don’t know. . .If I do use example files from another company, I always cite where I 
got that information [...].! guess the guideline I use is in the classroom situations, and it is in the educational 
settings (Participant 8). 

Another participant found it difficult to differentiate copyright and fair use in certain situations as well: 

There are times when you are trying to figure out just where the line is in the work. You know how close you 
are getting to the line where you are actually taking and not making different enough. And that’s gray area. I 
mean we try the best we can to stay in the law, but also we had to try to get our work done, you know, get what 
we need to the courses (Participant 2). 
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Another participant commented on the gray areas of copyright issues: 

There is not a clear cut, black and white issue. There is a lot of gray area, you know, just what is allowed, what 
isn’t. And you need to know a lot about every particular instance before you can say, well yes, I think you are 
right, I think you are not complying (Participant 18). 

In general, the majority of participants who considered copyright an ethical issue found it difficult to 
differentiate between copyright and fair use guidelines. The question then becomes whether technology helps rid of 
the gray area or makes it more complex? One participant noted: 

I think technology has got a long way to go, and there are still a lot of answers to. . .who owns this? Who has 
the ownership in the creation of that material? Let’s say I am in a team, and I am working on a project for a 
course I am working on. And I want to put this presentation report project, whatever it is we create as a team, 
in my portfolio. Well, who owns that project? Do I have to get permission from the other people in my team to 
publish that in my portfolio? And how do we check that? (Participant 19). 

Considering that the Digital Millennium Copyright Act of 1998 makes copyright more of a legal issue, we 
decided to further delve into the ethical aspects of copyright: Is it an ethical issue or a legal issue? Unanimously, 
participants said copyright was both a legal and an ethical issue. Comments from several practitioners listed below 
can give us a glimpse on the ethical aspects of copyright issue. One participant said: 

It is both. You don’t want to violate the law because it is law. On the other hand, I do believe that having done 
online work myself, I know how much work to the creation of those products. And I do believe that, creators 
have a right to earn some value of their work (Participant 13). 

Issue 2: Learner privacy 

With its swift expansion and growing popularity, information technology provides broad access to confidential 
resources (Pourciau, 1999). As a result, individual privacy and online confidential information are susceptible to 
being violated and abused. In this study, 13 participants cited that dealing with learner privacy was an ethical 
endeavor. One participant described how easily a learner’s private information can be misused with technology: 

And it makes easy for almost anybody to use it. There might be online grade book that you could cut and paste 
it into an Excel and publish it into a report and show it on-line. Message board, you can take a snapshot of it 
and paste it to a PowerPoint presentation and take it to a conference, and never consider the fact that in that 
snapshot you also have student names, you have student data (Participant 5). 

This participant continued to explain: 

Technology gives people a lot of capabilities that they might never have had before. And also along with that 
might open up possible opportunities for unethical behaviors that they may not even consider being 
ethical (Participant 5). 

In training situations, the potential of disclosing trainees’ private information to an unintended population can 
be an ethical issue to the participants. One participant described a situation in a training situation: 

The ethical kind of dilemma that we had is: when should we allow a staff member’s supervisor to actually see 
their scores? So that I think it is probably kind of an ethical dilemma that we face. When was it OK to do it? 
Did we need to give their permission? How far over the line where we are stepping with the privacy? [. . .] 
Because a staff member might not want their supervisor to know they are taking a lot of, you know, pretty 
detailed technology courses. For instance, if they are the office receptionist, or a secretary, and they are trying 
to move into web designer, or something. That person may not want their supervisor to know that they are 
planning on leaving because it might make their work environment a little awful for them. So that’s something 
we had to deal with (Participant 10). 

In general, technology makes individual privacy and confidential information accessible as well as vulnerable. 
In design and training situations, learner privacy can be violated in that learners’ online activities can be tracked, and 
learners’ private information, such as data and training scores, can be disclosed to an unintended population. 
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Issue 3: Accessibility 

Accessibility seems to be a particular ethical issue for learning technology practitioners working in an 
educational setting. The participants emphasized that accessibility means that a product reaches not only to a 
majority of its end users, but more importantly, to a small amount of users who have special needs, such as people 
with disabilities. One participant put it simply: 

Every new web site we create, we try to make it as accessible as possible (Participant 6). 

Another participant had similar concerns and provided more information: 

Whenever we promote something, we need to say in such a way that it is understandable, and it is relatable to 
all avenues of people and all sorts of life. Dealing with technological kind of function, you sort of speak in that 
language (Participant 14). 

Speaking about accessibility as an ethical issue, one participant explained: 

Accessibility is very important. We try to make things accessible if it is presented online [...]. This is where we 
can get into discussion on ethics. How far do we go down those lines? Does it really have to be accessible? Is 
that technology versus accessibility? We can really do something cool if we don’t care about accessibility, but 
we have to make sure it is 100% accessible (Participant 4). 

In summary, the participants indicated that making materials being accessible to a wider array of learners was 
more important than being ‘cool’ or using fancy technology. They emphasized that ethically speaking, it was 
the right thing to be more conscious of people with special needs 

Issue 4: Diversity 

Seven participants also cited diversity as a major ethical issue in their work activities. They indicated that 
diversity was an ethical issue both in design and training situations. In design situations, the participants pointed out 
that they had to be very conscious about the impact of online materials on people from diverse backgrounds. To this 
end, they made sure that content that they designed and produced with learning technology is not only 
understandable and meaningful, but also respectful to all avenues of people and all sorts of life. In training 
situations, diversity being an ethical issue was not only about taking consideration of people’s diverse backgrounds, 
it was also about not discriminating or stereotyping people when it comes to the use of technology, for example, 
discriminating against older people in seminars and workshops. 

Issue 5: Conflicts of Interest 

Technology training is taking place in many organizations on a daily basis. For this reason, people who have 
technological knowledge and skills are often outsourced by some external organizations. People who have access to 
computing resources are viewed favorably by external organizations as well. When learning technology practitioners 
take advantage of their skills and training materials that they developed on their work time to earn a second income, 
they potentially compete with their employer for business in an unfair way. Therefore, conflicts of interests occur. 
Four participants in this study recognized that they did experience conflicts of interest in their job, and that ethical 
decision making was involved 

Issue 6: Professionalism/Confldence 

The last ethical issue identified in the study, cited by three participants, was related to possessing sufficient 
credentials to perform confidently and professionally in design and training situations. The participants indicated 
that as technology skills become desirable in the workplace, many people from various disciplines and backgrounds 
suddenly engage in designing online materials or delivering technology training. Some practitioners may not have 
sufficient credentials to ensure quality of work. 



Conclusions 

In summary, six major ethical issues, in order of their frequency, were identified in this study: copyright, learner 
privacy, accessibility, diversity, conflicts of interest, and professionalism/confidence. 

Among these ethical issues, copyright, learner privacy, and accessibility have often been discussed in the 
literature with a focus on their legal connotations. This study, however, found that in online course design and Web- 
based training settings, these issues develop ethical ramifications beyond their usual legal consequences. 
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This study also identified three ethical issues that have not been discussed extensively in the literature: 
diversity, conflicts of interest, and professionalism/confidence. First, the identification of diversity as an ethical 
issue enriches the current literature. Previously, researchers proposed that the introduction of instructional 
technology can widen or close the inequality gap among different people and groups (Han, 1994; Hatcher, 2002). 
The results of this study suggests that diversity is an ethical issue in that professional technologists, who have a wide 
range of technical skills, can create learning materials that exclude people from diverse backgrounds and stereotype 
learners. Conflict of interest is another ethical issue that has not received adequate attention in the literature. The 
results of this study enrich Marby and O’ Driscoll’s (2003) discussions of how professional technologists use their 
knowledge and output outside of the scope of their jobs as well as identifying a few sources of conflicts of interest: 
taking learning materials developed on work time outside of the scope of work, taking advantage of the current 
position to earn a second income in other organizations, and using public resources for personal gain. The last 
ethical issue that is underrepresented in the literature is professional technologists’ professionalism and confidence 
in performing their work. The study found that it is unethical if professional technologists produce work of poor 
quality that results from only having knowledge and skill sets in either educational principles or technology 
applications. 

This study did not support several ethical issues that have been addressed in the literature. One issue is related 
to power and inequality, in which technology-based, career-enhancing activities can be accessed only by those who 
have power and resources. Another issue that is not supported by the study is the misuse of instructional technology. 
For example, if professional technologists overuse available technologies, they tend to ignore customers’ learning 
needs and replace existing learning methodologies. 

Recommendations 

As professional technologists face both possibilities and problems in the use of instructional technology, the 
identification of ethical issues is important to help establish effective guidelines for practice. With this 
understanding, our research and practice should be sensitive to not only who we are, but what we do, and why we do 
as professionals (Schwier, 2005). 

For Researchers 

The results of this study lay the groundwork for future study. Future research can continue to identify the 
ethical issues and coping strategies that help yield a clearer picture of ethical issues that professional technologists 
have experienced. Such information will be beneficial not only to our understanding of the ethical issues in the use 
of instructional technology in design and training situations, but also to help management provide possible strategies 
to handle these issues. Future studies can also be extended to investigate the nature of instructional technology, its 
maturity, and the assumptions that drive the technology by comparing our work with that of related professions 
(Schwier, 2005; Waters & Gibbons, 2004). Moreover, future studies can use different methodologies to investigate 
ethical issues and coping strategies such as quantitative methods or responses to hypothetical cases. Finally, future 
studies can explore the role of organizations and management in addressing ethical challenges posed to professional 
technologists. 

For Managers 

As the study revealed, the majority of the participants stated that they found it difficult to differentiate between 
copyright and fair use guidelines in educational settings. As a result, it is advisable that management not only covers 
copyright laws or guidelines, but also includes materials that apply these laws and guidelines to a collection of 
previous ethical situations. Another recommendation for management is to develop and implement a system for the 
reporting of ethical issues that have been encountered by professional technologists. Information from the reporting 
system can provide excellent cases that could be used to develop in-depth and skills-oriented seminars. Such 
information could also be used in employee orientations to socialize new professional technologists into the 
organization. Finally, as the findings in this study indicated, professional technologists turned to their team members 
and managers frequently for checks and balances when they were faced with ethical issues. Consequently, 
management should foster the two-way free flow of communication, which not only facilitates a smooth workflow, 
but also encourages informal sharing of the best practices of handling ethical situations. 

For Practitioners 

The results of this study also yield several recommendations for professional technologists. In relation to 
ethical issues, one of the first steps that a professional technologist can take is to locate relevant seminars, resources. 
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and experts in their organizations. They can also network with other practitioners in the field to share experiences so 
that their managers are not their sole source of reference. Ultimately, the professional technologists’ job should not 
only be limited to incorporating technology and instructional models into their complex array of responsibilities, but 
should also be extended to the awareness of the values they hold and the ability to use these values to inform 
practice. After all, similar to other professions, applications of instructional technology is not about a technical act; it 
is a process that includes moral and political consequences as a result of our choices for action (Schwier, 2005). 
Understanding the ethical implications of our choices will give us a grounding as we practice our craft. 
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Abstract 

This study investigated the effectiveness of the use of Computer-Mediated Communication (CMC) in 
learning pragmatics. The impact of teaching pragmatics by E-mail and WebCT discussion on Taiwanese EEL 
learners’ pragmatic competence was explored. Relative effectiveness of learning pragmatics through in-class 
activities and telecommunication were also compared. Data collected in school settings were analyzed 
quantitatively and qualitatively. The results showed that the pragmatic instruction enhance EEL learners’ pragmatic 
competence. 



Background and Theoretical Perspective 

Since the adoption of the communicative approach in second or foreign language teaching, more 
importance is given to the achievement of functional abilities in the target language. The development of second 
language pragmatic competence involves the ability to appropriately use a wide range of speech acts such as 
“greeting,” “apologizing,” “complimenting,” and “requesting.” Among them, requesting, the focus of pragmatics 
instruction in this study, has been one of the most studied speech acts. Various studies have also indicated that 
requesting is one of the most frequently used speech acts in communication (Ellis, 1992; Rintell & Mitchell, 1989; 
Rose, 1999). 

There is a general agreement that pragmatic knowledge in the second language can be acquired by utilizing 
universal pragmatic knowledge, and some aspects from the learner’s first language can be transferred to the second 
language. However, Bialystok (1993) has reported that in order to acquire processing control over the existing 
pragmatic foundations, adult second language or foreign language learners need to develop new representations of 
pragmatic knowledge not existing in their first language. In addition, research has shown that many aspects of 
pragmatic competence cannot be acquired without a focus on pragmatics instruction (Kasper, 2000). Therefore, it is 
reasonable to suggest that foreign language learning can be aided by instruction which helps learners practice their 
linguistic knowledge in communicative activities. 

In spite of existing theory and research evidencing the need for pragmatics instruction, English-as-a- 
Foreign Language (EEL) classrooms mainly focus on grammar-oriented instruction and pragmatic development of 
language learners has been overlooked. Studies have examined the fact that when pragmatics is not a planned 
subject in a second or foreign language classroom, the opportunities for developing pragmatic competence is quite 
limited (Kasper, 2000). The consequence is that English language learners who have studied English for years still 
face problems using language appropriately in communicative contexts. 

In addition, learning English is rather difficult in an EEL learning environment compared to an English-as- 
a-Second-Language (ESL) environment because EEL learners do not have the opportunity to interact with native 
speakers of the target language as ESL learners do. Language class activities in EEL settings often focus on 
decontextualized language practices, which do not expose learners to the types of sociolinguistic input that would 
facilitate pragmatic competence acquisition. 

Recently, interventional studies have examined the effects of explicit instruction in pragmatic competence 
on the development of learners’ pragmatic competence. The results from these studies have indicated the positive 
effect of pedagogical intervention, and this supports the view that pragmatic ability can be developed through 
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planned classroom activities (Bouton, 1994; Eslami-Rasekh, 2005; Rose, 1999; Takahashi, 2001; Tateyama, 2001). 
Studies conducted by Eslami-Rasekh (2004), Kasper (1997), Rose (1999), Takahashi (2001), and Tateyama (2001) 
also suggest that pragmatic features can be effectively acquired through explicit instruction on pragmatics. 

Over the past two decades, computers have become common instructional tools in the ESL/EFL classrooms. 
Currently, collaborative e-mail exchanges are one of the instructional tools used in classrooms. Studies have shown 
computer-mediated communication (CMC) has many merits in classroom settings. Computer-mediated 
communication refers to interaction via telecommunications. Electronic communication has been found to have a 
number of beneficial features that make it a good tool for language learning. Research has indicated electronic 
communication can enhance students’ motivation (Warchauer, 1996), and improve writing skills (Cononelos & 

Oliva, 1993). Cifuentes and Shih, (2001) further stressed that CMC provided an authentic context for learning 
functional abilities by having EEL learners interact with English-as-a-first-language speakers. With explicit 
instruction in how to communicate in the virtual environment, CMC may benefit the intercultural teaching and 
learning (Shih and Cifuentes, 2003). 



Objectives 

This study investigated the impact of pragmatic instruction on Taiwanese EEL learners’ development of 
pragmatic competence. Relative effectiveness of learning pragmatics through in-class activities and 
telecommunication were also compared. The present study attempted to answer the following research questions: (a) 
Did students who received the in-class explicit pragmatic instruction improve their pragmatic competence more than 
those who did not do so? (b) Did students who received the explicit pragmatic instruction through 
telecommunication connection from Texan tutors improve their pragmatic competence more than those who did not 
do so? (c) What was the relative effectiveness of learning pragmatic through CMC as compared to in-class 
pragmatic instruction? (d) What were students’ perceptions of learning pragmatics? 

Methodology 

This study applied a pretest-posttest control group experimental design and combinations of quantitative 
and qualitative data collection and analyses. The independent variable was the treatment with three different 
levels — (1) the control group which received no explicit pragmatics instruction, (2) the experiential in-classroom 
group which received explicit pragmatic instruction face to face from their classroom instructor, and (3) the 
experimental CMC group which received explicit pragmatics instruction from their Texan tutors through CMC (e- 
mail and WebCT discussion). The dependent variables were students’ pragmatic competence. 

Participants 

Participants were 82 undergraduate students majoring in applied foreign languages from a university of 
technology in Northern Taiwan. The other 13 participants were graduate students majoring in teaching English as a 
second language at a university from Southern Texas. 

In Taiwan, 82 students belonged to three intact classes and enrolled in the class of “English for Tourism.” 
Because of institutional constraints, it was not possible to assign students randomly to the different groups, thus 
making it necessary to work with three intact groups. In an effort to determine equivalence of the three groups in 
terms of their English language proficiency, the General English Comprehension Test was given to the participants. 
The statistics results showed that the control group produced higher mean scores on the reading comprehension 
pretest (M=3 1.067, SD=8.183) than the experimental in-classroom group (M=28.957, SD=7.258) and the 
experimental CMC group (M=28.966, SD=8.01 1). Nevertheless, three groups did not differ significantly from each 
other in the performance of the reading comprehension pretest (F=0.68, df =2,/>=0.51). 

There were 30 students in the control group, 23 in the experimental in-classroom group, and 29 students in 
the experimental CMC group. In Texas, each of the 13 graduate students was randomly assigned to be the tutor for 
two or three Taiwanese experimental group participants. These students interacted with their Taiwanese learners 
through email correspondences and WebCT discussion. All Texan participants received the instruction as part of 
their curricular activities in the class. 
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Procedure 



During the duration of this study (ten weeks), all eighty-two Taiwanese participants met once a week for 
one hundred minutes each time. At the beginning of each class, the professor in Taiwan spent fifteen minutes in 
dealing with class management and students affairs issues. Since the eighty-two Taiwanese participants were 
enrolled in “English for Tourism”, participants in all three groups were engaged in the following warm-up tasks: 
watching a short film about tourism in English for about fifteen minutes, followed by instructors’ explanation about 
film for about twenty minutes in each meeting. The instructor used the text book entitled: “At your service: English 
for the travel and tourist industry”. Each week, the instructor taught one unit of the textbook. 

During the remaining fifty minutes of the class, participants in the control group did not engage in any 
explicit pragmatics activities. Instead, the instructor spent about thirty minutes of lectures on learning tourism 
English using the teacher’s manual as a guide, followed by twenty minutes of summary and discussion in each 
meeting for a total of fifty minutes. During the thirty minutes of lectures, students had the opportunity to interact 
with the instructor through questions and answers. And students also had small group discussion with their peers 
during the twenty minutes of summary and discussion. Participants practiced their English in terms of writing, 
listening, reading and speaking during class. 

In contrast to the control group, during the remaining fifty minutes of the class, ten weeks lesson plans 
were delivered to the participants in the experimental groups; i.e., in-classroom and CMC. Each lesson plan 
consisted of one activity and each activity was designed in order to raise students’ pragmatic awareness. The content 
for both groups are identical and was based on the ten weeks lesson plans developed by the researcher. The 
components of the lesson plans aimed to raise students’ pragmatic awareness and offer learners the opportunity for 
communicative practice. For the experimental-in-classroom group, the lesson plans were delivered by the instructor 
through face to face mode; for the experimental -CMC group, the lesson plans were delivered though email 
correspondences and Webct discussion between Taiwanese students and their tutors. 

At the beginning of the study, all students were asked to complete the Discourse Completion Task (DCT) 
pretest. Students in the control group received the regular classroom instruction that did not explicitly address 
pragmatics in the teaching contents. The experiential in-classroom group received explicit pragmatic instruction 
face to face from their in-classroom instructor, and the experimental CMC group received explicit pragmatics 
instruction from their Texan tutors through telecommunication (e-mail and WebCT discussion). 

Following ten weeks of treatments, all students were asked to take the Discourse Completion Task (DCT) 
posttest. At the end of the study, the experimental CMC group took students’ perceptions of learning pragmatics 
survey to explicate their attitudes toward learning pragmatics, attitudes toward using e-mail and WebCT in learning, 
and perception of learning from Texan tutors. 

The Discourse Completion Task (DCT) included twelve situations with a special focus on speech act 
function of request. These situations were designed to probe how participants respond in different situation in terms 
of social status, power, and impositions. The social contexts specified in the DCT contain relationships between a 
professor and a student, a boss and an employee, and among friends. The purpose of this design was to see how 
participants interacted or responded to certain situations from different points of view. Two native English speakers 
rated the participants’ Discourse Completion Task (DCT) pretest and posttest productions. The rating system used 
in this study was adapted from the rating system proposed by Hudson, Detmer, and Brown (1995), containing 
components as the followings: (1) the ability to use correct speech acts, (2) expressions, (3) the amount of 
information, (4) levels of formality, (5) levels of directness, and (6) levels of politeness. In this case, the last three 
components were combined as one (levels of politeness) due to the overlapping elements of speech existing among 
these three components. The raters rated participants’ performance based on 5 point rating scale ranging from 1 to 5. 
The value for interrater reliability was reached to an acceptable level of agreement (r .90). 

Results 

The descriptive statistics results of the DCT pretest scores by group are demonstrated in Table 1. There 
were four scores, including the score of the ability to use correct speech act, the score of expressions, the score of 
the amount of information, and the score of levels of politeness. The two experimental groups overall yielded 
higher mean scores than the control group against three rating components (expressions, information, and 
politeness). However, the experimental in-classroom group scored slightly lower in the speech act rating component 
as compared to the relative means of the control group and the experimental CMC group. In this case, there was no 
significant group effect for the DCT pretest; namely, three groups did not differ in their pragmatic abilities in the 
speech act function of request prior to the treatment (F=2.131, df=2,/>=0.126). 
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Table 1: Descriptive Statistics Results of the DCT Pretest Scores by Group 



Rating Components 




Groups 






Control (N=30) 


In-Classroom (N=23) 


CMC (N=29) 


Speech act 


Mean 


46.00 


45.70 


48.45 




SD 


5.92 


5.65 


4.63 


Expressions 


Mean 


42.74 


44.09 


45.79 




SD 


5.77 


4.70 


3.89 


Information 


Mean 


45.28 


46.45 


47.00 




SD 


5.58 


4.72 


4.53 


Politeness 


Mean 


44.83 


47.09 


48.07 




SD 


5.76 


4.65 


5.32 



After treatments, the group comparison of the DCT posttest scores was conducted. The descriptive statistics 
results indicated that there existed greater discrepancy among the group means. The performances of the 
experimental in-classroom group and the experimental CMC group were better (surpassing from 2.7 points to 7.4 
points) than those of the control group in each of the four rating elements (See Table 2). 
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Table 2: Descriptive Statistics Results of the DCT Posttest Scores by Group 



Rating Components 




Groups 






Control (N=30) 


In-Classroom (N=23) 


CMC (N=29) 


Speech act 


Mean 


45.07 


48.32 


52.45 




SD 


4.43 


4.83 


4.44 


Expressions 


Mean 


40.52 


45.82 


47.66 




SD 


5.00 


4.60 


4.52 


Information 


Mean 


45.11 


47.82 


50.10 




SD 


3.69 


5.01 


4.24 


Politeness 


Mean 


45.19 


48.73 


50.83 




SD 


4.39 


5.49 


4.55 



The repeated measures MANOVA results further showed that there was a significant difference among 
three groups on the four means of the DCT posttest (F=16.35, df=2,/7<.05). The results of Tukey Honestly 
Significant Difference (HSD) Post Hoc test informed that the experimental in-classroom group and the experimental 
CMC group both scored significantly higher than the control group on the DCT posttest; whereas, the students in the 
experimental CMC group performed as well as those students who were in the experimental in-classroom group. 
Moreover, an interaction effect of the group by the four rating elements of the DCT posttest was found to be 
significant (F=2.93, df=6,/7=0.009). 

Figure 1 demonstrated the scores on expression yielded by three groups were significantly lower than the 
scores of other components (speech act, information, and politeness). The students in the experimental CMC group 
scored higher in the four rating components than the experimental in-classroom group, though difference was not 
significant. The control group produced significantly lower scores in all four rating components when compared to 
the two experimental groups. The experimental CMC group was found to have the highest mean scores on the 
speech act rating component, revealing their superior ability to use correct speech act than other elements necessary 
to make appropriate requests. 




GROUPS 

Figure 1. The interaction effect of the group by the four rating elements of the DCT posttest 

After ten week conventional pragmatic instruction, the experimental in-classroom group showed significant 
improvement in their DCT productions. Overall, the experimental in-classroom group generated significantly higher 
scores on the DCT posttest than the DCT pretest (F=l 1.156, df=3,/7<.05); the means for each rating component on 
the DCT posttest demonstrated an apparent increase, ranging from 1.37 to 2.62 points (See Table 3). 
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Rating components 


Tests 


Mean Scores 


Standard 

Deviation 


Speech act 


pretest 


45.70 


5.65 




posttest 


48.32 


4.83 


Expressions 


pretest 


44.09 


4.70 




posttest 


45.82 


4.61 


Information 


pretest 


46.45 


4.72 




posttest 


47.82 


5.01 


Politeness 


pretest 


47.09 


4.65 




posttest 


48.73 


5.49 



Furthermore, as Figure 2 showed the scores on expression yielded by the experimental in-classroom group 
remained the lowest scores whether on the DCT pretest or posttest. The experimental in-classroom group was found 
to have the highest mean scores on the politeness rating component; that is, participants tended to show diverse 
levels of politeness while making requests. Meanwhile, the mean scores on the speech act rating component 
displayed the greatest improvement from the DCT pretest to the DCT posttest (Mdct pretest-45.70, SDdcT pretest 5.65, 
Mdct posttest =48.32, SDDCTposttest=4.83). Other than that, the mean scores on expressions, the amount of information, 
and levels of politeness also fairly improved after the treatment. 




DCT 

Figure 2. The DCT pretest and posttest scores by the four rating elements for the experimental in-classroom group 

On the other hand, with ten week telecommunication connection to the Texan tutors, the experimental CMC 
group learned pragmatics through e-mail and WebCT discussion. The repeated measures MANOVA results showed 
a significant improvement of DCT productions for the experimental CMC group. Overall, the participants in the 
experimental CMC group generated significantly higher scores on the DCT posttest than the DCT pretest (F=47.897, 
df=3,/><.05); the means for each rating component on the DCT posttest increased ranging from 1.87 to 4 points. 

Compared with the performances in the DCT pretest, the experimental CMC group produced significantly 
higher scores (p<.05) on the DCT posttest in terms of four rating components. The mean scores on speech act 
displayed the greatest improvement from the DCT pretest to the DCT posttest (MDCTpretest=48.45, SDDCTpretest=4.63; 
Mdct posttest =52.45, SDDCTposttest=4.45) (See Figure 3). Other than that, the mean scores of the amount of information, 
levels of politeness, and expressions also fairly improved after the treatment. 
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DCT 

Figure 3. The DCT pretest and posttest scores by the four rating elements for the experimental CMC group 



Students’ Perception s of Learning Pragmatics On-line 

Students from the experimental CMC group mostly expressed that the explicit pragmatic instruction indeed 
helped them gain more knowledge in English pragmatics. Compared to the content presented in the conventional 
English reading or writing textbooks, the content of this pragmatic instruction was more practical and useful for 
their daily communication. For example, one student stated that he did not know the accurate meaning of “You 
rock” until his Texan tutor explained it to him. “You rock” in the United States meant “You are so cool”, and this 
student was pleased to learn more daily idiomatic expression in certain situations from his tutor. Another student 
also mentioned that he was strongly aware of the differences between Chinese and English pragmatics from 
communicating with his Texan tutor, and he regarded this learning experience as a valuable one because he hardly 
had the opportunity to interact with foreigners. 

Another student shared that the pragmatic instruction was beneficial to her for it helped her to make 
appropriate requests in the airport while traveling aboard. This student looked forward to learning more content 
with higher level of difficulty and more in depth because she thought the learning of pragmatics was quite useful and 
important. 

After ten weeks treatment, students in the experimental CMC group addressed that they enjoyed learning 
pragmatics by presenting examples for them in the first place, and then until they completely understood the content, 
they could use more examples to strengthen the concepts. 

Nevertheless, more than half of the students in the experimental CMC group responded that some English 
words and phrases used by their Texan tutors were not readily understandable; they had difficulty figuring out the 
meanings of certain messages. Several students in the experimental CMC group expected their Texan tutors to be 
more patient and affectionate. 

Students also reflected that the content of pragmatic instruction could be more situational, more animated 
with graphs, sounds, or short movies. A majority of students thought that the design of the content in the homework 
needed to be improved. The classroom instructor or Texan tutors should avoid posting similar and ambiguous 
questions every week, which bored the Taiwanese participants. 

One student pointed out that he felt frustrated, and exhausted when asked to write and type short formal 
essays independently per week, which was more like taking a formal serious English composition class. It was 
hoped that the content could be displayed from the easier level to progressively go up to the difficult level. 

Even so, most participants in the experimental CMC group were aware of the importance of pragmatics, 
and they realized that English was not as difficult as they thought before. They felt this learning experience was 
challenging, but interesting. It helped them gain more knowledge regarding Western people’s thinking patterns and 
writing styles. In addition, they also felt more comfortable when they used English to perform requests in contexts. 
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Educational Significance 



We found the pedagogical intervention had a positive impact on Taiwanese EFL learners’ development of 
pragmatic competence from this study. With the appropriate classroom management and the Internet access of 
computers, the students in the experimental in-classroom group and the experimental CMC group had the 
opportunity to engage themselves in the process of learning pragmatics. 

Additionally, computers functioned as “cognitive tools” for the experimental CMC group students to 
reflect, refine, and assess their structural knowledge. The findings of this study urge educators to integrate 
technology in helping Taiwanese EFL learners build up expertise in how to use English language appropriately, so 
that they can develop the ability to comprehend and generate productive communicative acts. 

It was apparent that the Taiwanese EFL learners did not naturally think and write in English. Accordingly, 
the Taiwanese students required more time to process the English textual information and to respond in English. If 
they were given more time on tasks, they might feel less concerned and threaten, and became more responsible for 
their own learning. 

Taiwanese EFL learners indeed need additional activities that can broaden their knowledge of pragmatics, 
and provide a broader variety of models and opportunities for them to supplement classroom setting of learning. We 
concluded that more complementary activities for EFL learners should be included in classroom settings, so that 
they can be given the opportunity to gain pragmatic knowledge. When pragmatics is explicitly taught to second 
language learners, they can acquire the essential skills faster (Bouton, 1994; Eslami-Rasekh, 2004). 
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An Investigation of the Effects of Multimedia Research on Learning 

and Long-Term Retention in Elementary Math: A Research Agenda 
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Abstract 

This paper first identifies that there is an underachievement in math nationwide as well as an underutilization of 
research principles from cognitive science in math curriculum and instruction. Then the paper provides a compelling 
rationale addressing the theoretical foundation and relevant prior empirical evidence supporting the proposed 
project. Finally, the paper provides the significance of the proposed project. 

1. Identify the Education Problem That Will Be Addressed by the Project 

(1) Underachievement in math 

Current levels of math achievement at the elementary and secondary levels within the 
U. S. suggest that we are neither preparing the general population with levels of math knowledge necessary for the 
twenty-first century workplace, nor producing well prepared students to meet national needs for domestic scientists 
and mathematicians (Research & Policy Committee of the Committee for Economic Development, 2003; U.S. 
Department of Education, 2003a, 2003b, 2003c). Despite the fact that levels of math achievement have improved 
over the past decade, achievement gaps continue to exist. Further, low levels of achievement are more likely to be 
present among minority groups and students from low-income backgrounds (U.S. Department of Education, 2003a). 
This underachievement in math may be related to math anxiety that is formulated in the later elementary grades 
(grades 3-6) with increasing exposure to events or significant others such as teachers and friends who solidify the 
anxiety (Hackworth, 1985). Additionally, there has been much debate regarding what constitutes math proficiency 
and which teaching methods support student achievement of this proficiency. Limited empirical research has been 
conducted to compare these instructional approaches to determine what leads to improved math achievement across 
ethnic and socioeconomic groups in the U.S. 

(2) Underutilization of research principles from cognitive science in math 

According to Carver and Klahr (2001), cognitive scientists have identified numerous basic principles of 
learning that are supported by empirical research in traditional, well-controlled laboratory settings. In math, over the 
past 20 years, cognitive and developmental researchers have described the growth of young children's scientific 
knowledge and numeracy 

(e. g., Kalchman, Moss, & Case, 2001; Lehrer, Schauble, Strom, & Pligge, 2001; Palincsar & Magnusson, 2001). 
However, for the most part, curriculum in math has not widely or systematically incorporated findings from this 
accumulation of research either through the creation of materials that support teaching and learning or at the level of 
instruction. Furthermore, little work has been conducted to evaluate the effectiveness of math curricula and 
instructional practices for improving student learning and achievement (Glaser, 2001). 

The three components of cognition (attention, memory, and reason) are the basis for achievement in math 
and other academic areas. Many scholars have called for research to bridge the gap between detailed, rigorous 
models of attention and successful academic performance (Baddeley, 1990; Carver & Klahr, 2001; Glaser, 2001; 
Hitch & Logie, 1996). Recent discoveries about multiple memory systems and their underlying neurological 
processes hold particular promise for developing instructional methods that tap into implicit learning (Baddeley, 
1990, 1999; Carver & Klahr, 2001; Hitch & Logie, 1996). Thus research is needed regarding how to improve the 
encoding of information and retrieval of implicit memories for learned material, how memory representations can 
better reflect what has been taught, and how the flexible use of memory systems can facilitate accurate reasoning 
and problem solving. 

2 . Provide a Compelling Rationale Addressing the Theoretical Foundation and 
Relevant Prior Empirical Evidence Supporting the Proposed Project 

(1) Provide a compellins rationale addressing the theoretical foundation 

According to Bork (1992), a multimedia learning environment is a new paradigm for learning in current 
education. This is especially true for math learning. According to Tooke (2001), “The computer affected 
mathematics education. It changed the mathematics curriculum, the teaching of mathematics, and even the way 
mathematics was leamed,”(p. 1). But investigating the effects of computer-based multimedia on learning and 
performance requires a solid foundation in cognitive psychology and learning theory (Astleitner & Wiesner, 2004; 
Dijkstra, Jonassen, & Sembill, 2001; Mayer, 2001; Rouet, Levonen, & Biardeau, 2001; Sweller, 1999). Multimedia 
scholars have recently built their research on cognitive learning theory (e. g.. Bishop & Cates, 2001; Schnotz, 2001; 
Wright, 2001). In addition, Mayer’s recent work of multimedia investigations are heavily grounded in cognitive 
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theory (Mayer, 1997, 1999; Mayer & Anderson, 1991, 1992; Mayer, Bove, Bryman, Mars, & Tapangco, 1996; 
Mayer & Gallini, 1990; Mayer, Heiser, & Lonn, 2001; Mayer, Mayer, Sims, & Tajika, 1995; Mayer & Moreno, 
1998; Mayer, Moreno, Boire, & Vagge, 1999; Moreno & Mayer, 1999, 2000a, 2000b, 2002). 

Mayer (2001) based the majority of his recent multimedia work on an integration of Sweller's cognitive 
load theory (1988, 1999), Paivio’s dual coding theory (Clark & Paivio, 1991; Paivio, 1986), and Baddeley's working 
memory (WM) model (Clark & Paivio, 1991; Paivio, 1986). Mayer focuses on the auditory/verbal channel and 
visual/pictorial channel. Mayer establishes his cognitive theory of multimedia learning through the following model 
(p. 44). 



MULTIMEDIA 

PRESENTATION 



SENSORY 

MEMORY 



WORKING MEMORY 



LONG-TERM 

MEMORY 




Figure 1. Mayer’s (2001) Cognitive Theory of Multimedia Learning 

This model is based upon three primary assumptions: (a) Visual and auditory information is processed through 
different information processing channels, (b) Each channel is limited in its ability to process information, (c) 
Processing information in different channels is an active cognitive process designed to construct coherent mental 
representations. Furthermore, this model is activated through five steps: ”(a) selecting relevant words for processing 
in verbal working memory, (b) selecting relevant images for processing in visual working memory, (c) organization 
of selected words into a verbal mental model, (d) organizing selected images into a visual mental model, and (e) 
integrating verbal and visual representations as well as prior knowledge,” (Mayer, 2001, p. 54). 

According to Mayer (2001), multimedia design is a potentially powerful system for enhancing human 
learning. The two approaches to multimedia design are: technology-centered approach and learner-centered 
approach. The technology-centered approach typically starts with the functional capabilities of multimedia and 
focuses on the incorporation of multimedia into emerging communication technologies. According to recent 
research (e. g., Cuban, 1986; Cognition and Technology Group at Vanderbilt, 1996a), the technology-centered 
approach does not usually lead to durable improvements in education. Thus scholars in multimedia research have 
focused on another important theoretical alternative— the learner-centered approach. The learner-centered approach 
typically starts with how the human mind processes information and focuses on how to use multimedia technology 
as an aid to enhance human cognition and learning. The learner-centered approach supports the idea that humans 
process information more effectively if the multimedia design is consistent with the way the human mind works. 

Based on Mayer and colleagues’ recent studies, Mayer (2001, p. 184) proposed seven multimedia design 
principles: (a) Multimedia : Students learn better from two or more types of media such as words and pictures rather 
than from words alone, (b) Spatial Contiguity : Students learn better when corresponding words and pictures are 
presented close to each other rather than far apart on the page or screen, (c) Temporal Contiguity : Students learn 
better when corresponding words and pictures are presented simultaneously rather than successively, (d) Coherence : 
Students learn better when extraneous words, pictures, and sounds are excluded rather than included, (e) Modality : 
Students learn better from animation and narration than from animation and on-screen text. Thus multimedia 
presentations involving both words and pictures should be created using auditory or spoken words, rather than 
written text to accompany the pictures, (f) Redundancy : Students learn better from animation and narration rather 
than animation, narration, and on-screen text. Thus multimedia presentations involving both words and pictures for 
the same content should present text either in written form or in auditory form, but not in both, (g) Individual 
Differences : Design effects are stronger for low-knowledge learners than for high-knowledge learners, as well as for 
high spatial learners rather than for low spatial learners. In all, this model provides instructional designers with a 
summary of main factors that have to be considered when developing multimedia-based learning environments 
(Astleitner & Wiesner, 2004). Many other cognitive researchers (e. g, Kumpulainen, Salovaara, & Mutanen, 2001; 
Leahy, Chandler, & Sweller, 2003) also suggest focusing on the design of instructional situations and pedagogical 
supports for multimedia-based learning. 
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Recently many multimedia scholars have suggested that all courses be redesigned on a national or 
international scale based on the characteristics of the multimedia environment (Bork, 1992; Confrey, 1996; Cuoco & 
Goldenberg, 1996). However, systematic multimedia integration into the current math instruction has yet to be 
integrated in the schools (e. g, Neuwirth, 2002; Robertson, 1998). The creation and development of a multimedia 
version of current elementary math curricula will follow Mayer’s (2001) multimedia design principles described 
previously. All content areas in the elementary math curriculum will be examined for adding appropriate multimedia 
components. The design and development of such multimedia curriculum will be based on the principles and 
standards for school mathematics from the National Council for Teachers of Mathematics (NCTM, 2000). The 
project will also be based on the Analysis, Design, Development, Implementation, and Evaluation (ADDIE) model 
(Dick, Carey, & Carey, 2001). The ADDIE model includes five components: (a) Analysis : Including analyzing the 
elementary students’ characteristics, learning tasks, and environments, (b) Design : Including writing course 
objectives, developing course outlines, and determining effective multimedia delivery methods for each content 
chapter, (c) Development : Including writing course content by integrating multimedia (e. g., video clips, animation) 
and sound into math instruction, such as content presentations, readings, activities, and assessments, (d) 
Implementation : Including delivering the multimedia instruction to the experimental student, (e) Evaluation : 
Including formative weekly or unit evaluation that is used to improve instruction during the multimedia instruction 
every week and yearly summative evaluation that measures the overall effectiveness of the course at the end of each 
semester and/or academic year. 

(2) Provide relevant prior empirical evidence supporting the proposed project 

The study of technology improving learning in the classroom is a debatable one, due to the relative novelty 
of using technology in an educational setting. In evaluating the effectiveness of technology on students’ cognition 
and learning, several factors must be considered. These factors include things such as the goal of the instruction and 
the information presented, the type of technology used, the implementation and use of the technology by the 
educator, and the students’ personalities. Most studies have found that these factors must be considered in order to 
determine if technology improves learning (e. g., Amone, 2003; Baker, 2000; Barton, 1998, 2002; Schwartz & 
Beichner, 1999). 

Mayer’s (2001) seven multimedia design principles (described previously) are consistent with the 
proposition described in the above paragraph. Based on Mayer’s seven multimedia design principles, Mayer and 
colleagues have conducted a series of studies investigating the nature and effects of multimedia presentations on 
human learning for the past decade. According to Mayer, there are two basic goals of multimedia learning. The first 
goal is to remember information. This refers to the ability to recall, recognize, or reproduce information presented. 
This goal focuses on retention of acquired information. The second goal is to understand information. This refers to 
the ability to apply acquired information to novel situations. This goal focuses on transfer of the acquired 
information. 

Based on Mayer and colleagues’ recent multimedia research, Mayer (2001) proposed three possible types 
of multimedia learning outcomes: (a) no learning at all, (b) rote learning, and (c) meaningful learning. In meaningful 
learning, students will learn integrated knowledge and exhibit good performances in both retention and transfer. 
Mayer and colleagues focused on investigating the effects of multimedia presentations on college students’ 
cognition and learning. Mayer further summarized the empirical evidence based on retention and transfer tests that 
he and his colleagues administered in their research. With the exception that no differences were found regarding the 
temporal contiguity effect for retention between both experimental and control groups, the effects of all principles 
were better for both retention and transfer in the experimental group than in the control group (p. 185-186). 

Recent limited studies have found that multimedia has a positive impact on learning across K-12 
educational settings (e. g., Felix, 1998; Gouzouasis, 1994; Khalili & Shashaani, 1994; Maddux, Johnson, & Willis, 
2001; Najiar, 1996; Scarlatos, 2002; Wenglinsky, 1998). These effects have been found in a variety of school 
subjects such as math (Mikk & Luik, 2003), reading (Heimann, Nelson, Tjus, & Gillberg, 1995; Passig & Levin, 
2000), geoscience (Chang, 2004), biology (Koroghlanian & Klein, 2004), instrumental education (Orman, 1998), as 
well as for special education students such as autistic students (Heimann, Nelson, Tjus, & Gillberg, 1995; Higgins & 
Boone, 1996; Tjus, Heimann, & Nelson, 2001), learning disabled students (Langone, Shade, Clees, & Day, 1999; 
Okolo & Ferretti, 1996), and dyslexic learners (Dimitriadi, 2001). In addition, Maki and Maki (2002) directly 
attributed improved learning to multimedia comprehension in online courses. 

Multimedia instruction has been specifically noted to be effective especially for learning abstract subjects 
such as math (Apostol, 1991; Apostol & Blinn, 1993; Blanchard & Stock, 1999; Clark & Paivio, 1991; Paivio, 1971, 
1986, 1991; Szabo & Poohkay, 1996; Tooke, 2001; Wiest, 2001) since teaching with multiple representations 
facilitates and strengthens the learning process by providing several sources of information (Grouws, 1992; Kozma, 
Russell, Jones, Marx, & Davis, 1996; NCTM, 1989). Multimedia technologies also have great potential as learning 
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activities in the primary school context (Baxter & Preece, 1999; Eskicioglu & Kopec, 2003; Jones, 1994; Nulden & 
Ward, 2002; Smith, 1999). According to Jones (1994), commercial multimedia software titles for children usually 
lack directed learning objectives. Research examining the impact of computer-controlled multimedia on student 
learning in school subjects is relatively recent, beginning around 1994. Jones encouraged researchers to focus on 
exploration of multiple simultaneous representations in school math. Now, 10 years later, there has been progress in 
this field. Based on recent literature, there are three basic types of projects investigating the impacts of multimedia 
in K-12 math instruction and learning. 

The first type of project includes projects involving the use of multimedia resources to train preservice 
math teachers at the university level (e. g.. Bitter, 1994; Bitter & Hatfield, 1994; Bitter & Pryor, 1996; Daniel, 1996; 
Dugdale, 2001; Garcia, 1999; Herrington, Herrington, Sparrow, & Oliver, 1998; Herrington & Oliver, 1999; Kim, 
Sharp, & Thompson, 1998; Lamport & Ball, 1998; Townsend & Townsend, 1992). Lamport and Ball (1998) used 
hypermedia to enable students to access teacher education materials in two classrooms videotaped over a one-year 
period. That is, the preservice teachers were video-taped while they were being trained on math instruction. Barron 
and Goldman (1996) focused on a major strand of math curriculum and used cases for preservice teachers to 
investigate and discuss. Herrington, Herrington, Sparrow, and Oliver (1998) designed an interactive multimedia 
resource to develop preservice teachers’ knowledge and practices for teaching and assessing mathematics in K-12 
classrooms. Herrington and Oliver (1999) described a qualitative study on students’ use of higher-order thinking as 
they use an interactive multimedia program based on a situated learning framework. Sharp and Kim (2000) studied 
six hours of multimedia-enhanced instruction over a two-week period for preservice elementary teachers in a 
methods course that emphasized constructivism and involved problem-solving multimedia. Their study found that 
multimedia-enhanced instruction facilitates students’ teaching abilities and confidence in their future teaching. 
Bowers and Doerr (2003) designed and investigated a multimedia case study for preservice mathematics teachers in 
methods classrooms. They found that the multimedia case approach received the highest ratings from students 
enrolled in courses where the case exploration activities were integrally woven into the course goals. Overall, most 
of these projects promoted students’ learning of math teaching methods and their teaching in other fields by 
integrating multimedia to situate the materials in real contexts. 

The second type of study includes students’ authoring and/or creation of specific multimedia projects. In 
the multimedia project conducted by Penuel, Golan, Means, and Korbak (2000), teams of K-12 teachers were 
trained to develop curriculum-based multimedia projects with students. Results indicated that students in the 
multimedia project classrooms consistently outscored their peers in the non-project classrooms in various areas such 
as understanding content, applying principles of designs, and adapting to meet students’ needs. Maor (2001) studied 
10^^ grade students to conduct a formative evaluation of an authentic data-based multimedia program. The students’ 
formative evaluation resulted in helpful information for modifying and improving the existing multimedia program, 
as well as raising further questions regarding how to promote students’ development of desired high-level thinking 
skills in the program. McLarlane, Williams, and Bonnett (2000) involved elementary students in a multimedia 
authoring project related to drug education using HyperStudio software. Neither the teachers nor the students had 
previous multimedia experiences, although they were provided with training. At the end of the study, various 
measures were used to assess students’ understanding of the content. The study indicates that student-produced 
multimedia projects can reveal important aspects of learning that traditional tests may not recognize, such as higher 
order thinking skills and processes. However, most of these studies were conducted either with middle school 
students (e. g., Liu & Hsiao, 2002; Macnab & Litzsimmons, 1999; Wilder & Schoech, 2002) or high school students 
(e. g., Koroghlanian & Klein, 2004; Looi & Ang, 2000). 

The third type of project includes designing and delivering some kind of multimedia instructional materials 
and then assessing their impact on students’ learning. The Cognition and Technology Group at Vanderbilt (1996a, 
1996b) developed a well-known series of films called the Jasper Series. The series utilizes hypertext terminology in 
order to describe their theory that their films offer semantically rich anchors which construct a macrocontext. These 
macrocontexts present meaningful authentic problems that must be solved. Smith and Westhoff (1992) describe the 
Taliesin Project that extends far beyond a simple program and speaks to the institutional context as well, including 
(1) curriculum revision with the aim of overcoming the segmentation of subjects and introducing interdisciplinary 
block teaching that covers math, science, and technology, (2) development of a hypermedia system with 
protocolling, communication, network connection, and application, and (3) development of tools for designing 
lessons. Johnson and O’Neill-Jones’ (1999) collaborative team designed a multimedia title called “Wyzf s 
Playground” to emulate and simulate the real life scenario of building a playground for 4^^ grade students to practice 
as a part of math curriculum. Results from this study indicate that the playground project supports the instruction of 
math and proficiency measures in five NCTM (1989) standards: spatial, patterns, computation, statistics, and 
communication. 
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Macaulay (2003) conducted an experiment to study the impact of multimedia on math learning 
performance of non-English- speaking third world children. In the study, 36 elementary students were randomly 
assigned into two groups: multimedia (experimental) and non-multimedia (control). All 36 students completed a 
pretest before the experiment and no statistically significant differences were found between experimental and 
control groups. Two types of computer-based learning applications on the same type of computers were used to 
present addition and subtraction concepts in elementary math. The control group was presented the materials using 
text only while the experimental group used a combination of text, images, animation, and sound. The presentation 
of math materials in both groups was similar in terms of structure and interactivity. The only difference between the 
two groups was the inclusion of multimedia components in the experimental group. Results from the posttest 
indicated that the multimedia group significantly outperformed the non-multimedia group in terms of the mean math 
test scores. 

Moreno (2002) conducted a study on who learns best in elementary math with multiple representations 
based on an interactive multimedia game in emigrant, low-income, Spanish-speaking children. According to this 
study, presenting symbolically and visually how an arithmetic procedure works in math does not ensure that all 
students will understand the explanations unless cognitive theory of multimedia learning is applied to the design. 
Thus this project suggests focusing on integrating multimedia into the math curriculum design by applying the seven 
principles from cognitive theory of multimedia learning described previously. 

Currently, computer-based instruction containing multimedia appears to be increasingly used as a 
supplement to traditional classroom instruction. However, systematic research investigating the impacts of 
multimedia-based learning in math is still largely lacking (Christian, 2003; Fan & Orey, 2001; Herrington, 
Herrington, & Sparrow, 2000; Koroghlanian & Klein, 2004; Royce, 2002). In order to stimulate more research in 
this field, an international cooperative workshop entitled “Multimedia Tools for Communicating Mathematics 
(MTCM)” was held in Portugal in 2000. The meeting aimed at providing, “fruitful and stimulating ground for the 
further development of multimedia tools for mathematical education, communication, and research,” (Borwein, 
Morales, Polthier, & Rodrigues, 2002, p. vi). Thus this project is designed to meet the aim of the international 
meeting as well as to meet other multimedia design challenges in math proposed by other researchers in the field (e. 
g, Astleitner & Wiesner, 2004; Cognition and Technology Group at Vanderbilt, 1996b). 

Based on the above review, it is found that the multimedia curriculum and instruction are neither a cure-all 
for problems facing the schools nor are they fads without impact on student learning. When designed and used 
properly, multimedia curriculum and instruction may serve as important tools for improving student proficiency in 
math and the overall learning environment of the school. This is consistent with Wenglinsky’s (1998) conclusion 
regarding the impact of computers on student learning. In addition, a careful review indicates that those limited 
studies in the literature are not systematic or rigorous for various reasons: (a) most multimedia designs did not 
systematically follow the principles based on cognitive science and proposed by Mayer (2001); (b) most research 
designs in elementary math were not well controlled; (c) most studies did not examine the variable of attention; (d) 
some studies that were highly controlled have problems generalizing to real educational classrooms; (e) none of 
those studies assessed the long-term effects of multimedia instruction on students’ standardized tests; and (f) no 
studies covered one or more years of elementary math curriculum. Thus this study is necessary to address all of the 
described previously challenges. 

3. Provide the Significance Of the Project 

Although many math teachers also use overheads in their teaching, the effects on student learning are still 
very limited. The most significant change in the classroom today is not just the addition of computers, but direct 
Internet access (Cattagni & Farris, 2001; Education Week, 2001; Williams, 2000). However, education has not 
progressed beyond the machines. For most students in K-12, the lecture/textbook/workbook approach is the primary 
medium for the delivery of math instruction. Unfortunately, textbooks are often presumptuous about the breadth and 
depth of students’ background knowledge, are written at readability levels that exceed those of average students and 
fail to structure information to facilitate comprehension of important concepts and relationships. As a result, students 
from low-income backgrounds are often excluded from receiving the benefits of effective math instruction 
(Chambel & Guimaraes, 2002). 

This project is designed to investigate the effects of multimedia research on learning 
and long-term retention in elementary math. Based on (a) the NCTM directives, (b) Mayer’s (2001) cognitive theory 
of multimedia learning, and (c) multimedia literature presented in the previous section, it is hypothesized that the 
multimedia math curriculum will have significant positive impacts on students’ cognitive processes and learning in 
math. That is, there will be significant mean differences between the experimental group (that will receive the 
multimedia math curriculum and instruction) and the control group (that will receive regular math curriculum and 
instruction) for low-income third graders in terms of these dependent variables: (a) attention to math instruction, (b) 
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immediate recall of the newly acquired math information, (c) long-term retention of the newly acquired math 
information, and (d) academic standardized test scores in third grade as measured by the math portions of the 
standardized achievement tests. 

The focus of this project is not on a replacement for math teachers, but using the computer-based 
multimedia curriculum and instruction to assist teachers and students in the math learning process. The actual 
intervention in this project is multimedia curriculum and instruction in elementary math. Multimedia-based 
curriculum and instruction can be an effective alternative to the exclusive dependence on textbook-based math 
instruction. In this project, multimedia refers to the integration of different media such as text, graphics, animation, 
sound, video (digital or analog), imaging, and spatial modeling into a computer system where appropriate (Jonasses, 
2000). Multimedia representations including animation typically provide a rich set of communication vehicles and 
have been essential for communicating and transmitting mathematical concepts to learners (Chambel & Guimaraes, 
2002 ). 

This project aims to foster meaningful learning by integrating multimedia design principles into the 
elementary math curriculum in grades 3-5. This project will examine how to design, develop, and deliver the 
effective multimedia curriculum and instruction by integrating multimedia design principles in elementary math 
curriculum materials for grades 3-5 in local elementary schools for three years. The planned curriculum materials, 
when fully designed and developed, will form an instructional sequence that covers the math curricula for grades 3- 
5. This project has three major tasks: (a) to design and develop effective multimedia curriculum and instruction for 
math based on recent cognitive research in multimedia and cognition as described previously, (b) to deliver the 
effectively designed multimedia curriculum and instruction for math, and (c) to empirically examine the effects of 
the above multimedia curriculum and instruction on students’ learning, cognitive processes, and academic 
achievement for low-income and low-achieving children. 

This project will advance solutions to underachievement in elementary math that teachers confirm to be a 
critical problem from their point of view. According to Eskicioglu and Kopec (2003), computer-based multimedia 
should be employed as the centerpiece for an emerging pattern of instruction. We agree with this perspective based 
on our recent teaching and research experiences. We have found that computer-based multimedia can promote 
independent and cooperative learning, and improve performance of low achievers and special student populations. 
The overarching purpose of this project is to bridge the gap among the laboratory research in cognitive science, 
multimedia learning, and the practices of in-service teachers. This knowledge could then be used to improve 
elementary math instruction. 

This project is significant in both practice and theory for many reasons. In practice, different from previous 
multimedia research projects, this project focuses on (a) one fundamental school subject— math, (b) the design and 
development of computer-based multimedia curriculum and instruction by integrating Mayer’s seven multimedia 
design principles mentioned previously, (c) numerous variables of students’ cognitive functioning such as attention, 
recall, and long-term retention, and (d) students’ academic achievement as measured by standardized tests. This 
project will also theoretically contribute greatly to the multimedia literature in K-12 math curricula and education. 
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Improving Learner Performance through Olfactory Intervention 

Aaron Loewer 



Abstract 

Sensory inputs in educational environments can influence learner performance. Specifically, the sense of 
smell (olfaction), with its neurological connections to emotion can be utilized to enhance learner performance and 
reduce learner anxiety. The purpose of this paper is to review aroma-based learning research in order to answer the 
following research questions: Can a pleasant ambient aroma enhance learner performance when released in an 
instructional environment? What is the influence that a pleasant aroma might have on learner anxiety? The results in 
most of the studies reviewed backed the use of pleasant aromas in learning environments. Two possible explanations 
are that some pleasant aromas have been shown to enhance connections between the learner and subject matter, 
while they also contribute to reductions in human anxiety. 

Introduction 

Olfaction, or the sense of smell, is a powerful, emotionally tied capability (Herz & Engen, 1996). This 
paper demonstrates that humans can learn to use olfaction to enhance performance in diverse environments and 
situations. Various behaviors or tasks exist in which dealing with olfactory information is a key element. For 
example, a car driver who perceives the odor of an overworked engine is presented with olfactory cues to change 
driving behavior, such as shifting to a higher gear, or to provide maintenance to the car by checking levels of 
radiator fluid or oil. The smell of worn out brake pads can also prompt behavioral or maintenance choices like 
shifting to a lower gear so the engine can slow the car’s momentum, or checking the thickness of the brake pads to 
see if repairs are needed. 

Food preparation is another area of activity that provides multiple instances where olfaction can aid 
performance. When selecting foods, chefs can use their olfactory sensitivity to perceive the ripeness of fruits such as 
pineapple and cantaloupe, or the freshness of milk or eggs. While baking or cooking, olfactory hints inform the cook 
as to whether or not an item has been cooked long enough, or too long. Prior to serving food, chefs use their sense of 
smell to evaluate food odor and taste as “perception of odors and flavors [has] similar neural bases” (Phillips & 
Heining, 2002, p. 204). Hence, the plugging of one’s nose can be an effective approach when taking in foods or 
medications that taste unpleasant. 

In a few cases, human behavior is unconsciously influenced by olfaction. One well-documented example is 
mate selection. Thornhill and Gangestad (1999) demonstrate that females not taking birth control pills are able, 
through olfaction, to detect males who will provide genes that will generate the healthiest possible posterity (p. 191). 
While males do not possess this ability, they do however possess the ability (along with females) to judge expected 
body scent attractiveness based on portrait attractiveness (1999). 

The influence that pleasant aromas can have on learner performance has been tested in numerous studies 
(Aggleton & Waskett, 1999; Baron, 1990; Baron & Kalsher, 1998; Cann & Ross, 1989; Deethardt, 2003; Diego, 
Jones, Field, Hemandez-Reif, Schanberg, Kuhn, McAdam, R. Galamaga & M. Galamaga, 1998; Ehrilichman & 
Halpem, 1988; Epple & Herz, 1999; Herz, 1997; Fudvigson & Rottman, 1989; Nagai, Wada, Usui, Tanaka & 
Hasebe, 2000; Smith, Standing & de Man, 1992). However, little is known in the instructional technology field 
about such findings. Consequently, some instructional technologists may be limiting themselves to designing and 
redesigning performance interventions without considering an important element of instruction — the use of olfactory 
abilities in triggering conscious and subconscious cues. 

The research questions for this review are two basic questions that instructional technologists need to know 
in order to effectively incorporate olfactory stimulation into their designs: Can a pleasant ambient aroma enhance 
learner performance when released in an instructional environment? What is the influence that a pleasant aroma 
might have on learner anxiety? The first question is important because designers need to know whether or not 
olfactory intervention even works, and whether or not it has been proven to work in various settings. The second 
question is based on the importance of emotions in olfactory research. As will be explained further, olfactory 
connections with the limbic system indicate that interventions involving pleasant aromas may reduce anxiety as 
experienced by learners. 
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Theoretical Basis 



Four principles explain the theoretical importance of reviewing studies that examine the influence of 
olfaction on learner performance and anxiety. Anxiety is defined by Merriam- Webster (2006) as an “abnormal and 
overwhelming sense of apprehension and fear often marked by physiological signs (as sweating, tension, and 
increased pulse), by doubt concerning the reality and nature of the threat, and by self-doubt about one's capacity to 
cope.” The four theoretical principles are discussed in the following order: environment, context, inputs, and 
affordances. 

According to Blanchard and Thacker (2006), the environment is one of the key factors determining 
human performance. Their model claims that performance is the product of three factors: (a) motivation, (b) KSA 
(knowledge, skills, and attitudes), and (c) environment. They explain “it is the combination of these factors that 
determines the person’s performance. The likelihood of engaging in any activity, then, is limited by the weakest 
factor (....) If the environment does not support the activity or blocks it, then it doesn’t matter how motivated or 
knowledgeable you are - you won’t do it” (p. 76). Granted, Blanchard and Thacker (2006) are speaking of more 
than just the sensory inputs in the environment, such as organizational structure and culture, but they are also clear 
regarding the importance of the surroundings in which performance is taking place. 

Environmental contexts that are increasingly similar during acquisition, retention and retrieval (Anderson, 
1995) can enhance performance during these neural activities. On the other hand, lack of contextual cues could lead 
to a decrement in performance. Driscoll (2000) states “When context changes from learning to application or 
practice, learners often fail to transfer the knowledge they acquired in one context to the other, related context” 
(Driscoll, 2000, p. 154). Furthermore, she states that “context plays an important role in learning. In familiar 
contexts, learners can relate new information and skills more easily to what they already know than if the learning 
context is unfamiliar” (p. 154). 

The problem of transfer can sometimes exist when learning environments or contexts are dissimilar from 
those in which recall takes place. High-end simulations can be employed to reduce contextual gaps, but for 
simulations to truly reflect all aspects of a leaming/performance environment, all sensory inputs should be 
considered. Gibbons and Fairweather (1998) emphasize that when representing information “the designer must be 
aware of the alternatives for communicating through all of the sensory channels: graphic and motion graphic, 
tabular, verbal and non-verbal auditory, tactile, kinesthetic, and olfactory” (p. 318). Making the point clear, Engel 
(2000) emphatically titles her book. Context Is Everything. 

The importance of communicating and representing information leads to the cognitivist concepts of 
“inputs” and “means” as discussed by Driscoll (2000). “The results of learning are to be explained through inputs 
(largely external to the learner) and means (largely internal to the learner)” (p. 156). Regarding inputs that are 
external to the learner, Gibson’s (1966) ecological perception theory introduces the concept of affordances. “The 
affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill” 
(Gibson, 1986, p. 127). In other words, “affordances are an object’s properties for which it can be used in a 
particular moment - they are an object’s invitations. For instance, a door handle possesses the initial affordance of 
grasping and, secondarily, turning” (Loewer, 2006, p. 24). Gibson (1986) summarizes: 

The perceiving of these mutual affordances is enormously complex (. . .) is based on the pickup of the 
information in touch, sound, odor, taste, and ambient light. It is just as much based on the stimulus 
information as is the simpler perception of the support that is offered by the ground under one’s feet. For 
other animals and other persons can only give off information about themselves insofar as they are tangible, 
audible, odorous, tastable, or visible, (p. 135) 

In summary, various theorists and practitioners emphasize the importance of four theoretical principles: 
environment (Blanchard & Thacker, 2000), context (Driscoll, 2000; Engel, 2000; Gibbons & Fairweather, 1998), 
inputs (Driscoll, 2000), and affordances (Gibson, 1966; 1986). While not necessarily mentioned in the studies 
examined in this review, each principle contributes to the theoretical basis for examining olfaction-based learning 
studies. Designers and developers who consider these principles during the ISD process may find grounds to 
produce instruction that incorporates olfactory cues along with inputs to other sensory modalities. 

Literature Review 

This review consists, first, of summaries of two previous reviews of olfactory research. Afterward, the 
review of individual studies that pertain to the research questions is presented. Two integrative reviews were located 
in which the relationship between cognition and olfaction was examined, these are Richardson and Zucco (1989) 
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and Herz and Engen (1996). Unfortunately, both pairs of reviewers failed to present their findings in a concisely 
summarized manner. 

Richardson and Zucco (1989) reviewed previous research on the following topics: Subjective description of 
odors, odor detection, odor identification, memory for odors, and brain damage leading to loss of olfactory ability. 
The reviewers did not include studies that examine the influence of olfaction on variables pertinent to this review 
such as anxiety, mood, associated memories and learner performance. Like many studies in olfaction, Richardson 
and Zucco (1989) focused primarily on the human abilities (or disabilities) of remembering, describing and 
categorizing odors. 

Herz and Engen (1996) reviewed previous research on the following topics: (a) memory for odors 
(including long and short-term odor memory), (b) verbal abilities to describe odors, (c) olfactory imagery, (d) 
distinctiveness of odor memory, and (e) memories evoked by odors that are autobiographical or context-dependent. 

Their review lends itself much more to the topic of this paper, in particular, summaries of research 
conducted on human memories evoked by odors. They summarized “the first direct examination of odor-evoked 
memory” (p. 306), as found in Herz and Cupchik (1995) where the researchers “used a paired-associated incidental 
learning paradigm to examine whether odors evoked more emotional memories than verbal cues” (Herz & Engen, 
1996, p. 306). They found that the odor and verbal cues were equal in their ability to elicit correct responses but that 
the odor cues had a “higher emotional intensity” (p. 306). 

Regarding context-dependent memory, Herz and Engen (1996) pointed to four studies in which an ambient 
odor is present during study and testing. These studies are Cann and Ross (1989); Herz (1997b); Schab (1990); 
Smith, et al., (1992). Herz and Engen (1996) also explained the cause of powerful emotions that are generated by 
olfactory experiences: 

The primary olfactory cortex forms a direct anatomical link with the amygdala-hyppocampal complex of 
the limbic system. Only two synapses separate the olfactory nerve from the amygdala, which is critical for 
the expression and experience of emotion (Aggleton & Mishkin, 1986) and human emotional memory 
(Cahill, Babinski, Markowitsch & McGaugh, 1995). Only three synapses separate the olfactory nerve from 
the hippocampus, involved in the selection and transmission of information in working memory, short- and 
long-term memory transfer, and various declarative memory functions (Eichenbaum, in press; 
Schwerdtfeger, Buhl & Gemroth, 1990; Staubli, Ivy & Lynch, 1984, 1986). No other sensory system makes 
this kind of direct and intense contact with the neural substrates of emotion and memory, which may 
explain why odor-evoked memories are unusually emotionally potent, (p. 300) 

Additional literature examined for this review included books, journal articles, web sites and personal vitae 
of researchers. Search terms included various combinations of the following: affect, anxiety, aroma, biofeedback, 
blood pressure, cognition, fragrance, heart rate, knowledge, learning, memory, nasal, nose, odor, olfaction, 
performance, perception, scent, sense, and smell. 

Databases for locating sources were accessed primarily through the library at Utah State University. They 
included: Digital Dissertations, Education Full Text, ERIC via EBSCO Host, ERIC via the US Department of 
Education, PsychINFO via EBSCO Host, and Biological Sciences. An associate accessed numerous fiill-text articles 
at the Jean and Alexander Heard Library at Vanderbilt University and sent them to the researcher. 

Inclusion Criteria 

The following were established as inclusion criteria for this review: Research must be conducted on 
recall of memories and/or learned content and skills, as opposed to research on recall of odor names, or odors 
themselves. This criterion alone caused the exclusion of the majority of articles cited in the aforementioned 
Richardson and Zucco (1989) review. Additional criteria included researcher documentation of: random assignment 
to treatment and control groups, odors used in treatment groups, odor release methods, sample size, and age of 
subjects. 

Originally, the researcher required documentation of effect size, but Herz (1997a) was the only 
researcher to include such findings, thus the criterion was adjusted. Although attempts were made to calculate effect 
sizes from results of other studies, the reported data was generally insufficient. Gathering other data was also 
unsuccessful. For instance, R.A. Baron was asked to provide data for items such as male to female participant ratios 
in Baron (1997), but he had “disposed of the data for that study a long, long time ago” (R.A. Baron, personal 
communication, December 6, 2005). 

The results in most of the studies reviewed backed the use of pleasant aromas in learning environments. 

Two possible explanations are that some pleasant aromas have been shown to enhance connections between the 
learner and subject matter, while they also contribute to reductions in human anxiety. 
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Sample 



Twenty-five articles were identified for inclusion in the review, many of which contained multiple studies. 
The studies were categorized according to the order of the research questions: (a) olfactory studies on learner 
performance, and (b) olfactory studies on learner anxiety. It should be noted that some of studies examined both 
performance and anxiety, thus, they were included in both categories. 

Olfaction and Learner Performance 

Olfactory studies on learner performance show that pleasant aromas such as peppermint, rosemary and 
lavender can be used to enhance connections between learning and testing. For example, Schab (1990) found that 
memory retrieval during a test can be aided if the same aroma is present in both the encoding and testing conditions. 
He also showed that learner performance can be further enhanced when the aroma is related, or congruent with the 
task at hand. For example, if a female student is learning about the biology of apples, her ability to recall specific 
content might improve if an apple scent is present during both encoding and retrieval. 

Similarly, Aggleton and Waskett (1999) found that congruent aromas distributed in a Viking museum 
enhanced participants’ ability to perform delayed recall of Viking-related content. They remembered “the types of 
clothing and jewelry worn by the Vikings, the types of foods eaten, the nature of the buildings and the classes of 
items sold in Viking York” (p. 3). 

Baron and Kalsher’s (1998) examination of the effects of a lemon fragrance on simulated driving 
performance is another example. Subjects were divided into four groups in which two treatments were 
administered — a lemon scent, or a gift, to randomly assigned groups. The researchers found that performance was 
best achieved in the scent condition in which no gift was received. 

Eighteen studies (mean sample size 58.7) that examined the effects of pleasant aromas on performance 
were included in this section of the review. They are: Aggleton and Waskett (1999); Baron and Kalsher (1998, 

Study 1); Baron (1990); Cann and Ross (1989); Deethardt (2003); Diego, et al., (1998); Ehrilichman and Halpem 
(1988); Epple and Herz (1999); Herz (1997a, Study 1); Herz (1997a, Study 2); Herz (1997b, Study 1); Herz (1997b, 
Study 2); Ludvigson and Rottman (1989); Nagai et al., (2000); Schab (1990, Study 1); Schab (1990, Study 2); Schab 
(1990, Study 3); Smith, et al., (1992). 

Twenty-one tasks were completed by participants in the 18 studies (Figure 1). The frequencies of these 
tasks were: Cognitive (8), Word recall (6), All others (4), Procedural (3). The most common aromas used in the 
performance studies (listed alphabetically) were almond, apple, chocolate, cinnamon, jasmine, lavender, lemon, 
peppermint, pine, and rosemary. 

In the 1 8 studies listed above, all findings were statistically significant in favor of using pleasant aromas in 
performance situations except two: Deethardt (2003), and Ludvigson and Rottman (1989). The Deethardt (2003) 
study examined participants’ ability to recall paired word associates where words were matched with specific 
aromas from seven combinations of dry teas, spices and incense. Ludvigson and Rottman (1989) had participants 
complete analogies, math problems, and vocabulary recall while being exposed to aromas of cloves and lavender. 
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Figure 1. Types of tasks completed by participants in performance studies. 
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Unfortunately, only two studies (by the same researcher) included a report of effect size, both of which 
used violet leaf in the scent condition. Herz (1997a, Study 1) reported an effect size of .1 1 when participants 
completed word recall tasks while experiencing public speaking anxiety. Herz (1997a, Study 2) reported an effect 
size of .42 when participants completed word recall tasks while experiencing test anxiety. While these effect sizes 
may not be large, they are effect sizes that show significant, positive results. 

The results of olfaction-based performance studies may be explained by Smith, et ah, (1992) who cite 
Tulving’s (1983) encoding specificity principle where “verbal memory performance is enhanced when the 
contextual, or incidental, stimuli present at the time of retrieval are the same as those which were present during the 
initial learning” (p. 339). 

Olfaction and Learner Anxiety 

As a result of direct links between olfactory processing and the limbic system (which some researchers 
claim to be the biological source of human emotional response), odors can strongly influence emotions. Kringelbach 
(as quoted by Hoelgaard, 2004) states that while other sensory inputs pass through the thalamus, the same “does not 
happen with olfactory input - that goes straight to the emotional response centre” (Tf 9). 

Olfactory studies on learner anxiety (indicated by both self-report and physiological data) show that 
subjects’ moods and attitudes can be influenced by both pleasant and unpleasant fragrances. For instance, Baron 
(1997) and assistants approached passersby in a shopping mall in areas where half of the passersby were in the 
presence of pleasant aromas (coffee shops, bakeries). The passersby were asked to complete a favor, either to 
provide change or the time of day for one of the assistants. Findings indicated “a higher proportion of passersby 
helped the accomplice when pleasant fragrances were present than when they were absent and that this was true for 
both” (p. 501) genders. Participants also filled out mood questionnaires (some before, and some after completing the 
favor). Subjects in the presence of the aromas reported being in a more favorable mood than subjects not exposed to 
the aromas. 

While the majority of olfactory researchers measure anxiety using self-report questionnaires, some also 
employ physiological assessments. Diego et al., (1998) used both self-report and EEG (electroencephalogram), also 
known as a brain wave test, to monitor subjects’ brain activity when given three minutes of aromatherapy. Two 
aromas, lavender (which is considered to be relaxing) and rosemary (stimulating) were used in the experiment. Data 
gathered using EEG showed that the lavender-exposed group experienced increased drowsiness while the rosemary- 
exposed group experienced increased alertness. These data were also consistent with subjects’ self-report of mood. 
Both groups performed math computations (pre- and post-aromatherapy) and results showed that both groups 
performed faster after the aromatherapy. Interestingly, the lavender group was more accurate in its math responses 
than the rosemary group, suggesting that relaxation may assist concentration (p. 223). 

Nagai et al., (2000) examined the influence of participant-selected aromas (and their documented 
intensities) during a psychomotor task. They used a blood pressure monitor and a polygraph system which 
monitored “respiratory movement, electrocardiogram, pulse wave of the left middle finger, and skin temperatures of 
the forehead and the left forefinger” (p. 228). Results indicated that during a rhythmic handgrip exercise, diastolic 
blood pressure was significantly lower for the odor group than the non-odor group. 

Twelve studies (mean sample size, 93.1) were included in this section of the review. As mentioned earlier, 
some overlap existed between the performance studies and anxiety studies, thus some experiments were included 
twice. They are: Baron (1990); Baron (1997); Baron and Kalsher (1998, Study 2); Baron and Thomley (1994); 

Diego et al., (1998); Herz (1997a, Study 1); Herz (1997a, Study 2); Herz (2004); Lehmer, Eckersberger, Walla, 
Potsch and Deecke (2000); Ludvigson and Rottman (1989); Sanders, Diego, Fernandez, Field, Hemandez-Reif and 
Roca (2002, Study 1); and Spangenberg, Crowley and Henderson (1996). 

Eighteen variables were examined in the 12 studies. Results in 13 of the 18 variables examined confirmed 
that pleasant aromas could reduce human anxiety. The non-significant affects came as a result of two aromas 
investigated: Lemon (1) and Violet leaf (4). Interestingly, a lemon scent was used in Baron and Kalsher’ s (1998, 
Study 1) simulated driving performance study, finding lemon to significantly enhance performance yet in Study 2, 
the lemon aroma did not positively influence participant affect. 

From the total number of affects reported, the frequencies at which specific types of affect were examined 
included the following: affective state after study (3), mood (3), emotional arousal (2), pleasure (2), relaxation (2), 
state anxiety (2), depression (1), emotionality of odor-evoked memories (1), affective evaluations of a store’s appeal 
(1), and trait anxiety (1). The most common aromas used in the twelve studies (listed alphabetically) were lavender, 
lemon, orange/orange citrus, rosemary and violet leaf 
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Amidst ongoing research, little is known of the affective domain. Martin and Reigeluth (1999) state that 
they “believe there are compelling reasons for including affective development in all types of learning 
environments” (p. 507). Goleman (1994) summarizes possible reasons for the importance of affective research and 
instruction. 

[The] architecture of the brain gives the amygdala a privileged position as an emotional sentinel, able to 
hijack the brain. [Sensory] signals from the eye or ear travel first in the brain to the thalamus, and then - 
across a single synapse - to the amygdala; a second signal from the thalamus is routed to the neocortex - 
the thinking brain. This branching allows the amygdala to begin to respond before the neocortex, which 
mulls information through several levels of brain circuits before it fully perceives and finally initiates its 
more finely tailored response, (p. 17) 

One group that is exemplary in affective examination is the Affective Computing Group 
(http://affect.media.mit.edu/) at Massachusetts Institute of Technology, which describes its work in the following 
way: 

Our research develops new technologies and theories that advance basic understanding of affect and its role 
in human experience. We aim to restore a proper balance between emotion and cognition in the design of 
technologies for addressing human needs. (][ 1) 

One of their current projects challenges learners to “focus on a topic such as math in the presence of strong 
emotions (problems at home, or feelings for the teen across the room)” (http://affect.media.mit.edu/projects.php, ^ 
6). The objective is to teach students how to apply meta-affective skills for handling emotionally heightened 
situations. 

Another branch of affect research promotes physiological data gathering in order to assess emotional 
conditions of learners. Moss (2003) asserts that heart rate, respiration, muscle tension and hand temperature are 
manifestations of anxiety, all of which can be physiologically gathered. As mentioned previously, some of the 
studies reviewed in this paper documented the use of physiological anxiety assessment (Diego et al., 1998; Nagai et 
al., 2000; Sanders et al., 2002). Peek (2003) explains that in gathering physiological data, instrumentation is used to 
“(a) monitor a physiological process of interest, (b) measure (objectify) what is monitored, and (c) present what is 
monitored or measured as meaningful information” (p. 45). As such, increasingly accurate information can be 
gathered from research participants, whereas self-report mechanisms may be unreliable, or the actual reporting by 
the participant may be biased. 



Conclusion 

One of the purposes of this conference is to strengthen connections between new technologies and 
educational goals. Where the goal of educators in most sectors is to create learning experiences that enhance learner 
performance, new technologies employed to achieve such a goal may be found in the domain of physiological 
assessment. If one of the first objectives of a designer is to understand the audience then physiological data can 
provide valuable details about learners, their possible levels of anxiety, and their experiences in various instructional 
environments. 

The concepts of environment, context, inputs, and affordances, as previously discussed, demonstrate the 
importance of strengthening connections between acquisition, retention and retrieval (Anderson, 1995) through 
olfactory intervention. Incorporating olfactory interventions into the “selection, sequencing and posing of problems” 
(Foshay & Gibbons, 2003, p. 22) may augment an instructor’s ability to foster contextual connections in learners. 

A sample of the growing body of olfactory research literature has been presented in this review. 
Specifically, literature has been evaluated regarding the influence of olfaction on learner performance and anxiety. 
Twenty-five articles, some of which contained multiple studies, were located and included in the sample. Overall, 
many studies point out the fact that not only do olfactory stimuli serve as adequate cues for recall, they also serve as 
potent cues for recall of highly emotional memories. 

To directly answer the research questions, the majority of the studies reviewed show that pleasant aromas 
can enhance learner performance. Additionally, many of the studies reviewed have also shown pleasant aromas may 
decrease human anxiety. The level of influence that aromas had on anxiety was not examined, but it is reasonably 
safe to conclude that various pleasant-smelling aromas may assist learners with relaxation during encoding and 
recall. Without such assistance, these experiences may remain cognitively, or affectively uncomfortable to learners. 
While this review focuses primarily on olfaction as an emotionally-tied human sense (Herz & Engen, 1996), there is 
much that the field might also gain through additional research that shows how olfactory stimulation might be 
combined with other sensory stimulation to improve performance and decrease anxiety. 
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Abstract 

Voices of Children is an innovative research project that explores the notion of multi-literacies through 
images and text produced by children from around the world. In the initial phase of the project, children from five 
different countries (Australia, Hong Kong, Malaysia, Thailand and the U.S.A.) have constructed photographic 
images and written responses that document their everyday experiences. The images and text are powerful mediums 
for communicating the personal and cultural dimensions of the children’s lives and, as such, provide a basis for 
analysis that draws on a range of different approaches. This paper provides a discussion of the framework for the 
analysis as well as presenting some important themes that examinations of the images and text have revealed. 

Overview of the Project 

A photograph is both a pseudo-presence and a token of absence. Like a wood fire in a 
room, photographs — especially those of people, distant landscapes and faraway cities, 
of the vanished past — are incitements to reverie (Sontag, 1977, p.l6). 

This paper reports on the findings of Voices of Children, an international research project that provides an 
opportunity for exploring meaning through photographic images and text constructed by children from across the 
world. This project was initiated by researchers from the Faculty of Education at the University of Wollongong 
(UOW) in Australia and was awarded funding through an international grant aimed at strengthening partnerships 
with institutions overseas. The initial phase included colleagues in four other institutions: Hong Kong City 
University, Universiti Sains Malaysia, Assumption University in Thailand and the University of Northern Illinois, 
U.S.A. Generous funding from Adobe Systems, U.S. later supplemented the original funds which, more recently, 
has supported the involvement of a number of other countries including Samoa, Guatemala, Zambia, Qatar as well 
as indigenous Australian communities located in remote areas. 

So far, over one hundred and fifty children have been provided with disposable cameras that allow them to 
take photographs of what they see on a daily basis. At the same time, they have been given booklets that encourage 
them to write about themselves, their families and the world in which they live. The images and text produced by the 
children are regarded as artefacts that reflect important personal, cultural and social dimensions. They complement 
one another and provide the viewer with opportunities for exploring common elements as well as diverse 
experiences in the lives of children across the world. The project also involves a travelling photographic exhibition 
that draws on the images and text as artworks generated by the children. Exhibitions in each of the countries 
involved in the project provide a platform for participants as well as a wider audience to examine the artworks as 
representations of individual and cultural narratives. An additional feature of the project is the development of a 
website that will give participants and others an opportunity to view the images. The website which can be found at 
www.voicesofchildren.org contains all of the images grouped according to the location from which they were 
drawn. 



The project encourages children and adults alike to make sense of the conditions that define their daily 
lives, as well as the lives of others. Its focus embodies a number of related perspectives: social justice, educational 
and pedagogical. From a social justice perspective, those who take part in the project are provided with opportunities 
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for engaging with the experiences of others and for empathising with them, a vital process where self-awareness and 
social change are goals. As Frank (2000) observes, “taking the other’s perspective is a necessary step in constructive 
social change” (p. 94, cited in Denzin & Lincoln, 2005, p. 668). Educationally, the project provides opportunities for 
children to make meaning of their real world experiences through mediums that encourage critical engagement in 
learning (McGirr, 2001). It supports children from different cultures in developing the skills they require for 
communicating, understanding, translating and critically examining their different worlds. Learning that develops 
multi-literacies requires strategies that provide for these deeper levels of meaning. Finally, from a pedagogical 
standpoint, educators must develop strategies for teaching and learning that support their students in developing a 
critical understanding of their world. Roslyn Arnold (2006) acknowledges the imperative of teachers to blend 
student experience and interests with the knowledge and skills inherent in effective learning, whilst responding to 
changing societal expectations and needs. Through the promise it offers of engaging students whose experiences are 
both similar and yet diverse. Voices of Children provides a useful basis for developing pedagogies that have 
relevance for this century. 



Introduction 

From an educational perspective, it is important to provide a multi-modal approach to teaching and 
learning, so that students are given opportunities for learning in a variety of modes and through a range of 
technologies (Unsworth, 2001). Educational interventions that focus on the development of multi-literacies support 
the ability of students to access and make meaning across a range of learning environments. In its efforts to keep 
pace with the rapid change that permeates the modem world, education has broadened its base and the simple 
concept of literacy has diversified to meet the demands made by changing curricula. Participation in this new 
learning environment requires skills that have not necessarily been provided by traditional educational practices and, 
where access to them may have been available, the skills required have not always been adequately developed 
(Brown & Bamford, 2004). Opportunities to explore meaning through images and text reflecting authentic 
experiences require children to develop a broader range of skills than has previously been the case. 

The combination of image and text as complementary dimensions within a framework for analysis are a 
unique feature of Voices of Children. Images often reveal experiences that do not translate into the written word, 
whilst that which is not readily visible may be revealed in writing. In both cases, image and text are the result of 
what Muffoletto (2001, p.2) refers to as “ideologically formed intentional acts” that enable viewers to constmct their 
own meanings. They provide opportunities for exploring personal, cultural and social experiences. In this project, 
the focus on visual literacy refers to the ability to make meaning from images, whilst the written responses provided 
by children clearly encourage an ability to engage with the written word (“literacy” in its most traditional sense). 
According to Kress and Van Leeuwen (1996), students need to think critically about visual data and develop the 
ability to understand layers of meaning. This viewpoint is equally as important for the written word. Importantly, for 
researchers, educators and students, the combination of detailed information provided through both avenues makes it 
possible to examine the personal lives of children as well as raising their voices, individually and collectively 
(Hauser, Brown & Prosser, 1997). Whilst globalisation has effected a blurring of the boundaries that define us 
(Weisner & Lowe, 2005), Voices of Children provides a valuable opportunity for preserving and valuing distinct 
identities as they exist across a variety of contexts. 

The development of skills across multi-literacies is vital for elementary and secondary school students to be 
able to engage in a meaningful way with the wide range of material with which they are regularly confronted in both 
formal and informal learning environments (Hubbard & Greh, 1991; Christopherson, 1997; Roblyer, 1998; Roblyer 
& Bennett, 2001; Brown & Bamford, 2004). The development of this knowledge demands the creation of a new 
language, a wider range of strategies and a diverse range of materials that will encourage students to make meaning 
through inter-modal interactions (Unsworth, 2001). Traditional educational practices and the strategies on which 
they frequently rest often fail to adequately and effectively support the learner in the development of the skills 
necessary for operating in an environment that is increasingly demanding. In response to this situation, opportunities 
that allow students to engage in meaningful experiences involving a range of literacies are vital (Muspratt, Luke & 
Freebody, 1997). 
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Research Framework 



The research underpinning this project seeks to involve children in experiences that will allow them to 
present their own views of their realities with limited adult direction. Their written responses have been generated 
by a range of simple statements asking them for information about their families, their experiences and their wishes 
for change in the world. With regard to the photographs they construct, they have been given the freedom to capture 
images without adult intervention or censorship. Through this combination of activities, the researchers have sought 
to provide as much scope for expression as possible. In his discussion of the value of audio-visual records for 
research in the social sciences, Kellehear (1993) observes that visual and other sources “can be important clues to 
the cultural world of people whose communication emphasis is not strongly placed on speaking” (p.73-74). In this 
case, “speaking” can be read as “writing” and, given the constraints involved when younger children as well as 
young people with limited experience in the English language are required to produce written responses, the value of 
the visual image is extremely important. 

In order to respond to the demands of this research, a number of key questions have been formulated to 
guide the project: 

1 . How do young people in a variety of different locations across the world represent their everyday 
experiences through photographic images and text? 

2. What words and phrases do young people use to describe and explain their experiences? What 
understandings of their experiences can we, as researchers, draw from their words? 

3. How do the images they construct reflect their experiences and what are the understandings we 
can draw from them? 

4. How can images and text combine to provide detailed accounts of the lived experiences of young 
people? 

5. What are the understandings that participants as well as audiences develop in response to the 
presentations of images and text as part of the Voices of Children project? 

6. How can the information gathered through this project be used to develop pedagogies that will 
address societal needs and expectations for educating young people from a range of different 
cultures? 

Our responses to these questions rest on a clear analysis of the data gathered through the project as well as 
a commitment to presenting images and text for audiences in ways that respect the integrity of the material that is 
produced by the children involved. No clear theoretical framework that encompasses the broad dimensions of this 
project is available and, instead, a grounded theory approach has been taken in the collection and analysis of data. 
This involves an iterative process that includes asking questions of the data in order to develop more detailed 
insights and rests on a reciprocal relationship between data collection, its analysis and theory (Strauss & Corbin, 
1990). In terms of analysis of both the images and text, methodologies that are appropriate to each medium have 
been considered. In both cases, content analysis has acted as a practical means of organising and categorising data, 
whilst a narrative approach with regard to interpretation has also been useful. 

Content analysis is a useful tool for organising and categorising the range of responses that the children 
have submitted. It provides a means of refining the data down into particular words or elements of the images. This 
method of analysis supports the identification of categories that include common as well as unusual responses 
(Silverman, 2001). Categories that are drawn from the data contribute to the identification of patterns across 
responses and, with the data organised in this manner, it becomes possible to identify emerging themes (Kellehear, 
1993). Interpreting the data from a narrative perspective enriches this process, making it possible to develop the 
themes in a more coherent way (Chase, 2005). 

Both images and text may be viewed as vehicles for conveying stories from a narrative perspective. 
Definitions of narrative vary enormously according to Rimmon-Kenon (2006) and Squire (2004) notes that, at 
minimum, a written response must contain two clauses in order to be considered narrative. Many of the written 
responses submitted by the children readily exceed this requirement. With regard to images. Banks (2001) suggests 
that internal and external narratives should be considered. Internal narratives embody the story that an image 
communicates to a viewer, whereas external narratives reflect the social context in which the image is embedded. 
According to Banks (2001), elements of an external narrative, or “information about the nature of the world beyond 
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the photograph, are always involved in readings of the internal narrative” (p.l2). Whether the stories are read 
through images or text, both contain narrative properties. White (1981, p.20, cited in Rimmon-Kenan, 2006, p.l4) 
suggests that our fascination with the stories of others occurs because, in them, “reality wears the mask of a 
meaning, the completeness and fullness of which we can only imagine, never experience”. Rather than representing 
transparent windows through which the lives of others can be viewed, narratives provide a means of gaining some 
insight into the perspective of another, allowing us to develop a taste of the experiences of others, at least in our 
imaginations. 



Reading the Images 

As global boundaries become less clearly defined, the image provides the basis for a common language. It 
is significant that for each of the children, as artists, the locations they occupy personally and culturally determine 
what they record (Ewald & Lightfoot, 2001). On examination of the images we find that they have been culturally 
determined through the choice of the artist. According to Fuery and Fuery (2003), “to comprehend this particular 
capacity and operation of the image, and its high significance in culture, images need to be ‘reviewed’ — that is, seen 
outside their normal functioning” (p.90). It is intended that the way in which the images are viewed will challenge 
and disrupt the perceptions and understandings of the audience. From this perspective, the project allows both the 
artist and the viewer the ability to describe and identify the way in which culture represents or expresses itself 
(Fuery & Fuery, 2003). 

Deconstructing or “unpacking” the elements of an image for analysis is a complex process. In this project. 
Collier’s (2003) approach has been adopted. This involved a collective overview of each country involved, 
systematically working within and between the images under review. Collier’s (2003) four-stage model, which 
incorporates the tenets of content analysis, made it possible to uncover a number of categories, patterns and themes 
and to develop them in detail as the relationships among them were explored. The particular categories that emerged 
include Context and Focations, Social Relations, Social Distance/Body Pose, Activities, Interaction, Spatial and 
Structural, Cultural forms and Intent. For the purpose of this paper, only one of these categories will be discussed in 
detail. 

Context and Locations 

All of the children provided a range of localities in their images. Spaces included indoors and outdoors, as 
well as spaces that could be identified as specifically “owned” by the children. Overwhelmingly, children included 
images of their bedrooms. These ranged from bare-floored communal sleeping sites inside Zambian homes to 
designer bedrooms in Australia and the U.S.A. The images ranged from tiny snapshots of bookshelves containing 
important trinkets to panoramic views of the rooms. Images of family members looking relaxed were provided, 
sometimes set in the kitchen, the lounge room or other locations inside and outside the home. Family photographs 
were generally posed but taken in informal settings. The images of Australian children (both urban and remote) 
involved large stretches of natural environments — ^bush landscapes and open areas (for example, note the image in 
Figure 1). This was in contrast to children from other countries who typically presented landscapes in which figures 
(family and friends) obscured the environment. Schools, exteriors and interiors of classrooms featured in most of the 
images, with the exception of those submitted by children from Zambia. 
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Figure 1 




Reading the Text 

Whilst the images in the project hold tremendous visual power, allowing an exploration of culture (Ball, 
1997), the written word provides a complementary dimension that can be read alone or in conjunction with the 
images. The children were invited to produce written accounts of their immediate worlds as well as providing a 
commentary of their global concerns, incorporating their experiences at the personal as well as the cultural level. 
Michael Crotty (1998) notes that written responses can be read as narratives when they incorporate personal stories 
as well as their cultural locations. As Crotty (1998) observes, a personal story presents “the voice of our culture — its 
many voices in fact — that is heard in what we say” (p.64). Rather than simply providing captions for their images, a 
requirement that may have limited both the images they captured as well as the written responses they constructed, 
the children were requested to provide personal information about their immediate environments, their families and 
what they enjoyed doing. In addition, they were asked to comment on global issues, providing opportunities for the 
development of critical thinking. 

Currently, one hundred and eighteen booklets have been returned with all but four containing written 
responses. The written responses were initially categorised by location and responses to the six questions included in 
the booklets were scrutinised on this basis. In addition, responses were also categorised on the basis of age and 
gender. With regard to age, at the times when their responses were submitted, the ages of the participants ranged 
from 6 to 18 years. Responses were categorised according to one of three sets of age ranges: 5 to 8 years, 9 to 12 
years and 13 years and above. Participants were also identified as either male or female. The written responses were 
coded using key terms drawn from the data and, as analysis across locations progressed it became possible to 
identify various patterns, with broader themes emerging as more data became available. Some brief comments will 
be made regarding general response patterns across location, age and gender before an outline of the emerging 
themes that have been identified is presented. 

General Response Patterns: Location, Age and Gender 

These three categories, location, age and gender, whilst useful for grouping responses, are not entirely 
independent of one another and their intersections are often informative. Nevertheless, based on these categories. 



267 




patterns have emerged with regard to the ways in which participants have written about their experiences. The 
following summary provides a brief account of some of these patterns. 

In terms of location, a child living in a remote community will have different access to communication, 
transport and leisure facilities than a child of similar age living in a bustling city. Access to these facilities shapes 
interactions with others as well as the nature and range of activities in which it is possible for the child to engage. 

For example, where daily experiences are less dependent on the natural environment, the use of technology assumes 
increasing significance. Children from the more remote communities in Western Australia and Zambia made limited 
reference to the ways in which technology played a part in their lives. Only one youngster in Western Australia, for 
example, mentioned watching movies on television. This is not to say that technological advances are not important 
to these communities but it may reflect the degree to which children in remote areas have access to items such as 
computers, televisions, video games, the internet and hand-held telephones. On the other hand, these items are 
mentioned frequently by children in urban Australia, Malaysia, Thailand and the U.S.A. 

When age is included as a basis for examination, patterns emerge within age groups. For example, there 
are common elements to the ways in which young people aged 9 to 12 years interact with one another, regardless of 
location. The nature of their leisure time and their interests in sporting activities attest to common pursuits. On the 
other hand, a difference in age-related skills was evident with regard to literacy, with shorter responses generated by 
children in the younger age group and longer, more detailed responses given by older participants. Other differences 
were also evident from a developmental perspective. For example, children in the 5 to 8 years age group tended to 
respond in ways that reflected a marked degree of egocentrism, whilst in the two older groups respondents were 
more likely to recognise and understand the perspective of others. A clear example of this difference can be found in 
the words of a young child living in a country town in New South Wales, Australia who, in response to the question 
about what he might change in the world declared: 

'7 want a McDonalds 

This somewhat hedonistic statement stands in contrast to the response of a child in the 9 to 12 years age group living 
in the U.S.A. who wrote: 

'df I could change anything in the world it would be to make sure everyone had enough money to 
have clean food, water, medicine, clothing, and a home. I read that around the world one billion 
people don ’t have these things. These people are suffering from disease other countries have 
completely eliminated. '' 

The older child can understand and empathise with people on a more global scale, assuming a perspective that draws 
on a range of knowledge and experiences not yet possible for the younger child. 

When responses are categorised with regard to gender it is again possible to identify differences as well as 
common elements. For example, both girls and boys refer to the importance of family and friends in their lives and, 
in the majority of cases, they refer to the roles of parents and, at times, themselves in traditionally gendered terms. 
With regard to difference, whilst both groups acknowledge their involvement in informal as well as organised and 
competitive sporting activities, a higher proportion of males in NSW, Australia and Zambia responded with 
reference to these activities than did females in these locations. In these two locations, all of the boys mentioned 
their engagement in sport compared with less than half of the girls. In addition, there were some differences in the 
sports that both groups identified. Girls, for example, were more likely to mention non-contact sports such as dance 
and gymnastics than boys were. 



Emerging Themes 

A number of themes have emerged from an analysis of the written data so far and, in certain instances, they 
share common ground with themes emerging from the images. These themes continue to be shaped as other 
questions are asked of existing data and as new data becomes available. At this point three themes are noteworthy in 
that each demonstrates different dimensions of the lived experiences of the children who have taken part in the 
project. A brief overview of these dimensions is included here. 
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A sense of place and space: Shaping the contours of experience 

A sense of place and space is evident in the responses that have been gathered so far and, in some respects, 
it shares elements of the visual images discussed earlier in terms of Context and Location. It embodies the ways in 
which experiences are shaped on the basis of the sense of place and space that the individual develops in relation to 
where he or she lives. 

Tom Anderson (2003) writes about the ways in which differences between the constructed or built 
environment and the non-constructed or natural environment shape our development. He argues that the lives of 
many people in the world are lived in constructed environments and that the sense of space we each develop “is 
mediated by our constructed milieu. . . [to the extent that, in our separation from the natural environment] . . .our own 
constructions become our world” (Anderson, 2003 ,p. 6). Both images and text reflect this viewpoint, with written 
responses from young people in remote environments suggesting a greater affinity with their natural surroundings 
than those who live in urban areas. On the other hand, young people living in built environments write confidently 
about the constructed landscape in which they negotiate their daily lives. 

Sharp contrasts were evident between responses from children living in a remote indigenous community in 
Western Australia when compared with children of similar ages living in urban and inner city areas. One of the 
children from Western Australia describes her home in the following terms: 

“Near the great sandy desert is Kadjina community and that’s where I live. The sand is red and 
the weather is hot. The grass is prickly and called Spinifex. ” 

Her words evoke the sense of space, the colours and the conditions that prevail in her home territory and, in 
conjunction with responses to other questions, it becomes clear that the environment shapes many daily experiences 
of the children living there. The activities the children undertake together as well as with their families reflect a 
connectedness with the natural environment. Another child writes: 

“My mummy and daddy are really good cooks. My daddy is a really good hunter. One day he got 
9 goannas. ” 

Catching goannas is never an easy task and one that children in built environments are not likely to experience! By 
contrast, a young boy living in Bangkok, Thailand, provided a description of his home that focussed more readily on 
built structures as landmarks than features of the natural environment: 

“My home is at [formal address specified]... 7//ve in the village near a hospital, school, market 
and department stores. There are almost 10 temples near my place. ” 

Half of the respondents from Thailand included their formal addresses as descriptors and the activities they 
described, which they typically shared with family and friends, were more likely to involve elements found in built 
environments. For instance, they describe activities such as bike riding in a park or going shopping with friends, or 
else playing sports that require access to particular facilities and equipment that may be less available in remote 
communities housing a smaller number of people. 

A brief description of a home in Zambia presented another view of how lives may be lived against a 
background of structures quite different to those mentioned by children in other locations. One boy wrote: 

“I live at a small place called [name of company], a ten acre area with few boss houses and a big 
compound with about 20houses for workers. A factory is in it fenced with electrical wire. ” 

The frugal nature of life in a compound within a relatively sparse landscape is borne out by references to the 
importance of a vegetable garden and livestock in other responses. 

Whilst all of the written descriptions provide the reader with a sense of place and space, accompanying 
images present a rich background against which interpretations of these responses may be made. 
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A sense of others: Developing empathy and compassion 

An overwhelming number of responses, irrespective of location, age or gender, in relation to the question 
“What’s good in the world” involved a sense of compassion for those in need. Not only was there an 
acknowledgement of suffering and the impact, in particular, this may have for other children but there was also 
recognition that one of the good things in the world involves people helping other people, irrespective of the 
boundaries that commonly define different groups. More detailed responses were provided by young people in the 
two older age groups but even the responses of the youngest children reflected a sense of altruism. 

A number of responses mentioned unusual events that had caused devastation, such as the tsunami in south- 
east Asia and Hurricane Katrina in the U.S.A., and the relief that was provided on a world-wide basis. Other 
responses indicated that children and their families were aware of and, in some instances, involved with local groups 
who responded to the needs of those in their own communities. At a personal and global level, the importance of 
helping others was viewed as a priority. 

With regard to recognising the plight of children in different areas of the world, a participant from Penang, 
Malaysia stated: 

'7 will stop child labour in India and Pakistan ...I would also stop the war and starving of 
children in Africa and Iraq... children become victims of war and suffer the most... People living 
in these countries also become victims of HIV and AIDS and again children suffer the most. ” 

Other responses from children echo the sentiments involved in helping those in need, often with a hope that those 
who cause suffering to others can be changed from “bad people” into “good people”. 

A sense of powerlessness: Living in fear 

Responses to the final question regarding their wishes for changing the world, led many children to write 
about the dangers they perceive, whether from terrorism, drug dealing or war, as well as their desire for peace. For 
some, the call for an end to war was couched as a plea, at other times a command, but a common element of the 
responses was the fearfulness this topic invoked, reflecting also perhaps the sense of powerlessness a child 
experiences in a world of adults. A young child from Bangkok, Thailand stated quite simply: 

'‘There are bad people in the world. So, we live in fear. ” 

A direct experience of war was evident in the response of one young girl living in Penang, Malaysia, who 
described the change she has encountered after leaving her country of origin, Iraq, to live in a safer environment. 

Her story continued in terms of how she might change the world: 

“There are three things that I would like to change in this world. First, to stop any war especially 
in my country (Iraq) and stop the killing of children. Second, no more poor people. Third, to build 
a big building and put orphans in it and take good care of them. ” 

The simplicity of her statements belies the depth of her first-hand experiences of war and its effects. A response 
from a child in Illinois, U.S.A. also demonstrates the fearfulness of the situation when war becomes personal: 

“If I could change anything in the world it would be for peace on Earth. Not just that but that the 
war in Iraq could end because my cousin is there and I am very scared for him... ” 

With regard to conflict and the tragedy involved in it, a young girl from NSW, Australia stated the case 
quite clearly and offered a solution: 

“Human conflict is the biggest problem in the world. If humans weren ’t so ignorant then there 
wouldn I be wars with millions of innocent people being murdered because of their race. Maybe 
this can be helped by education because if people come to learn about others they may come to 
appreciate them. ” 
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Children recognise the dangers inherent in criminal activities and in the war. They appreciate the implications for 
others and, where they are touched personally by these experiences, they can explain their fears quite rationally. 

Conclusion 

Voices of Children highlights the ideas and experiences of children across the world, integrating personal 
experiences that are culturally located with basic technology that is readily accessible. The images and text they 
have constructed are representative of them as individuals and also of their surroundings and relationships. They are 
empowered through the choices they are able to make in terms of the ways in which they present their experiences 
and, at the same time, their responses provide expressions of their particular cultural perspectives (Prosser, 1998). 
Additional features of this project such as the travelling exhibition and the website honour the contributions of the 
children as artworks whilst providing a range of ways in which the participants and others can engage with the 
images and text that have been submitted. The voices that emerge through this project reflect the many dimensions 
of children’s lives, their common experiences and concerns as well as the conditions that divide them. Importantly, 
as a collective, their voices reflect a promise for the world we all share. 
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Abstract 

iPods have become a cultural phenomenon. Not only do they serve as a mechanism by which to identify 
the Net-Geners, Millennials, and now post-Millennials (nicknames given to today’s students), but they also hold 
valuable promise as a means by which to reach students bombarded by fragmented learning environments, 
multitasking, and a world in which people are always connected. This paper discusses the research results and 
analysis on using Podcasting in curriculum delivery and focus on two issues, 1) the impact of Podcasting technology 
on student learning, and 2) course design issues involved in instructional delivery via Podcasting. 

Introduction 

Finding innovative methods to reach students is becoming increasingly difficult and even challenging. 
Today’s students, aptly dubbed the “Net-Gener” by leading scholars, are inherently more sophisticated than we were 
at their ages and have pushed the “convergence” catchphrase to the limits of its very definition - the coming 
together of various media types. In the Fall of 2005, various units within the University of Houston (hence UH) 
joined forces to begin a thorough research study investigating the benefits of Podcasting in the distribution of 
curricular content and understanding the efficacy of such tool in the modem classroom. This research project was 
based largely on the feasibility that iPod can be an effective portable learning tool to enhance student learning. 
Capable of on-demand media delivery. Podcasting made the connection between education and students via this 
portable learning device. 



The Study 

A survey method was used to conduct this study for the purpose of analyzing student perception and 
behavior associated with instmctional delivery via Podcasting. Phase I of the study found four classes participating: 
one section was delivered completely online; the remaining sections utilized WebCT Vista as an online course 
management system. The total enrollment in these four classes was 200. Nearly 80% (N=165) of the students 
participated in the study. Instmctors used Podcasting to deliver course lecture materials or supplement materials as 
alternative channel to facilitate student learning. The study specifically investigated the areas discussed below 
related to Podcasting of course content materials and use of portable devices as leaning tools. 

There is no question that iPods have become as commonplace as the cell phone in our student’s lives. In 
fact, the simple exercise of observing students walking between classes or sitting outside on benches or in other 
social areas across campus reveals little white ear buds everywhere - an obvious sign of the time. In addition to 
offering students an efficient method for listening to music, podcasting allows educators a unique opportunity to 
meet students in their own, comfortable environments. And these opportunities do not only exist in the various types 
of lecture and/or addendum content that we can make available, but also exists in three basic categories of podcasts 
as well. 



The success of phase I of the UH Podcasting pilot and research study yielded valuable data surrounding 
student’s use of iPods, Podcasts, and perceptions associated with usage. Looking closely at both functional and 
ritualistic applications of iPod use, students at the School of Communication at UH generally preferred iPod over 
other MMDs, and though the ratio of iPod to other MMD use was nearly three to one, there was no significant 
difference in how students used the various devices to consume podcasting content, as well as music and other 
media types. 
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Using iPods and other MMDs to Access Instructional Materials 

iPod users clearly dominate the sample. Student reported using iPod more than other MMDs, with 69% of 
them preferring iPod to other MMDs. While generic mp3 players comprised in insignificant proportion of the 
overall MMD population, it is important to note that, of the 69% of the students who had iPods, that the type of iPod 
used by students varied to some degree (Table 1); there was no real winner, though it was impressive to note that a 
fair amount of them had 5^^ Generation video iPods, which helped in the analysis of Vodcast (video podcast) use in 
instructional delivery. 

Table 1 Frequency Table of Different Models of iPod Owned by Students 



iPod Model 


Frequency 


Percent 


Cumulative Percent 


Fifth Generation iPod 


23 


23.2 


23.2 


iPod nano 


25 


25.3 


48.5 


iPod with color display 


2 


2.0 


50.5 


iPod photo 


2 


2.0 


52.5 


iPod shuffle 


11 


11.1 


63.6 


iPod click wheel 


4 


4.0 


67.7 


iPod mini 


17 


17.2 


84.8 


iPod touch wheel 


13 


13.1 


98.0 


iPod scroll wheel 


2 


2.0 


100.0 


Total 


99 


100.0 





Student iPod / MMD Usage per Week 

Most of the students in the pilot reported that they used their iPods and MMDs 20 hours per week or less, 
with almost 37% using iPod just five hours per week, nearly 19% using theirs between five and ten hours per week, 
and 24% between 10 and 20 hours per week respectively. The preferred time and location that students used iPod 
was almost as varied as the number of options available for them to identify. The data indicate that students found a 
wide variety of opportunities to use their iPods: whether at home, driving in the car, or exercising, students were 
almost evenly split when it comes to preference {Figure 1). This suggests that their podcasting use in education 
could leverage the benefits of the device’s mobility with the habits of use by students. 

Figure 1 Student preferred time to use iPod (in percentage) 
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Preferred Activities with iPod / MMDs 



Listening to music is the major use of iPod among students. But, let us not forget the original intent for the 
little devices: they were, after all, originally conceived for music organization and mobile listening. Technology has 
advanced enough in just five short years to allow the viewing of both images and now video - a major advancement 
in mobile media device delivery. Though most certainly the best is yet to come, one cannot ignore the amazing 
potential that iPod brings to the classroom and lecture hall. While only 7.5% of those surveyed reported they 
preferred using their MMDs for academic content delivery, the fact that most still preferred using them for music is 
not surprising or even unexpected. 



Podcasting Course Materials 

After reviewing student preferences with respect to overall use of iPod and MMDs, we can now examine the 
academic use of podcasting and student perceptions and behaviors surrounding such use. 

Methods for Podcasting Playback and Consumption 

Whether at the gym or on the beach, there are no real clear-cut winners in the various activities where 
students use their iPods or MMDs. However, when students are using iPods or MMDs for academic purposes, it is 
clear that podcasting makes contribution in the manner in which students are studying. Though viewing podcasts 
and other curriculum materials on the computer represented the majority of consumption methods, downloading 
academic podcasts to MMDs became an increasingly popular activity over 31% of the time (Table 2). 

Table 2 Methods Students Used to View Podcast Course Materials 



Methods to View Podcasts 


Frequency 


Percent 


Viewed materials on a computer 


108 


62.8 


Downloaded materials to iPod 


54 


31.4 


Downloaded materials to other MP3 player 


6 


3.5 


Burned materials to CD for playback 
on a portable CD player 


4 


2.3 


Total 


172 


100.0 



Time Spent Using iPod or MMD with Academic Podcasts 

Though students preferred using iPod and other MMDs primarily for music, they overwhelmingly 
supported using these devices for consuming academic podcasts and other class materials. Even with general 
approval, it is interesting to note that more than 75% of those who used podcasting content on their iPods did so for 
five hours or less per week. This would suggest that students reviewed and consumed podcasting content primarily 
on the computer, but used the iPod as a means to review or consume on the go, offering mobility as an option for 
extending the delivery channel from the constraints of the computer. 

Of the 127 students who said they used the iPod or MMD for reviewing or consuming academic podcasting 
content, over 17% did so between five and ten hours per week. This is a significant number which suggests that a 
large segment of the population found the additional delivery channel to be an effective resource in lecture or 
addendum content delivery. Additionally, less than 5% of those who used the iPod for this purpose did so ten or 
more hours per week, with nearly 1% doing so between 20 and 30 hours weekly. While this may seem extreme, 
those who commute via public transportation (or other methods) could easily find that time period optimum for use. 
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Perceived Quality and Preferred Podcasting Formats 

Perhaps one of the most often overlooked components of podcasting is the need to generate quality content 
from the outset. Given that students will likely be consuming academic podcasts in a variety of settings, it is 
imperative that they be produced in a manner that will allow for efficient delivery while preserving resources. 
Because podcasts are actually downloaded to the users computer and ultimately to the iPod, file size and 
compression can make the difference between a 5-minute download and a 25 -minute download seem like an 
eternity. One of the biggest challenges that those who encode and publish podcasting content face is trying to apply 
the appropriate compression and encoding settings so that consumers find the minimal download time. One problem 
with this, however, is that the greater the compression and encoding, the greater the degradation of the final product. 
This is especially challenging for video playback and consumption, especially for academic content that involves 
instructor examples, such as math, accounting, statistics, or other classes where it is important that the student be 
able to see playback clearly. 

Figure 2 Student perceived quality of podcasts in different format 




It was extremely important in this pilot, from the beginning, that quality of content was addressed and 
evaluated throughout the process - and, in most cases, before students were exposed to the content themselves. To 
accommodate theses activities, a Quality Assurance team was included in the organizational structure of the pilot 
with the primary responsibility of monitoring both publishing methodologies as well as playback quality of all 
podcasting formats. Knowing, then, that we had quality product from the outset meant that we were able to focus on 
unadulterated feedback from participants who might have otherwise been biased because of poor quality playback. 

Feedback from students suggests that audio-only podcasts were thought to have the highest quality, with 
video coming in a close second. Enhanced casts rounded out the three, though all formats received relatively high 
marks for quality. While the question posed was initially meant to gage their perception of the quality of the 
product, it is worth mentioning that students may have associated quality of the podcasts with the preferred format 
for consumption. 

Clear and Effective Instructions 

One of the most important components to Podcasting is providing effective and adequate instructions for 
consuming academic content through podcasting. These instructions should include, at the very least, the URL 
(feed) for subscription to the podcasts, as well as alternate options for receiving the content. In addition, the 
instructions should most certainly be made available electronically, through course management tools or other 
electronic syllabi. Given that the medium is produced and consumed online, making the instructions for consuming 
them available in the same medium is highly recommended. In this study, student feedback on the instructions given 
was collected for effectiveness and future improvement {Figure 3 & 4). 
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Figure 3 (left) The instructions to access the podcast course materials were clear 
Figure 4 (right) The instructions to access the podcast course materials were easy to follow 




□ Strongly Agn00 

□ Agr00 

□ M'A 

■ Disag [100 

□ Strongly Disagr00 




□ Strongly Agr00 

□ Agr00 

□ N/A 

■ DisagrGG 

□ Strongly Disagr00 

■ Missing 



The instructions used in phase I of the pilot were standardized for the most part, and students were given 
the same instructions for each class, with only feed URL being customized for the instructor of record. The 
overwhelming majority of those polled responded that the instructions by instructors were both easy to use and 
understand, as well as extremely helpful in the subscription process. 

Reasons for Using Academic Podcasting 

One of the greatest challenges we face as educators is ensuring that our students have equal access to 
content, even when those students cannot always be in class. Podcasting can be a tool to help in this process, 
providing a mechanism for students to gain access to materials when they otherwise would not be able to do so. Like 
the times and places that students prefer to use iPods, the reasons that they used podcasts are almost as equally 
universal. Having no clear preferred reason, students found benefits in all classifications made available to them for 
selection {Figure 5). 

Figure 5 Student reported reasons to view podcast course material 



One of the important aspects investigated in this study was to find out student perceptions towards 
educational use of Podcasting. The above-mentioned data indicated that majority of students are using iPods or 
MMDs for entertainment. Delivery course material through Podcasting is still relatively new to students. Their first 
podcasting experiences are very important for instructors to integrate this technology effectively. The following 
group of data shows the effect of podcasting on student learning experiences from different perspective. 

Podcasting and Learning Styles 




Student Perceptions and Experiences with Podcasting 
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Learning Style Theory has been considered as an important factor for developing effective teaching 
strategies and methods. Since Podcasting can be presented to students in different formats, audio, enhanced, and 
video, this technology can help instructors respond to student diverse learning styles by creating rich learning 
environments that engage students with auditory as well as visual learning styles. In this study, the data indicate that 
there are more than 43% of students self-reported as visual learners, and this group is followed closely by 41% of 
tactile learners. Auditory learners take up 15% of the total number of students. When students were asked which 
podcasting format best suits their personal learning style, video podcasting (visual learners) takes the lead with close 
to 60% of respondents. The data provide valuable feedback for instructors on podcasting delivery format. 

It should be noted that even though Vodcasting is favored by students in this study, in reality, the 
production and playback of Vodcasting have higher requirements for both instructors and students as compared to 
audio podcasting. For instructors, it involves video taping, editing, and encoding, etc., it could be very time- 
consuming process; and for students, they need portable devices that can display video images. The format selection 
should be determined by the content analysis. If audio format can satisfy student learning, there will be no need for 
instructors to go through the lengthy process. But the data are very good indications for future podcasting 
development when technology are advanced to the point where requirements will not be so demanding for both 
instructors and students. 

Podcasting and Study Habits and Patterns 

The availability of technology affects people’s life. We can see people’s living habits and behavioral 
patterns changing with the development and advancement of technologies. For example, since online education 
became available, quite a few students no longer attend regular on-campus classes. They access learning content and 
interact with their instructors and classmates virtually. Podcasting technology has created a new instructional 
delivery channel. In this study, there are more than 32% of participants reported that the availability of course 
material podcasts changed their study habits; and about 30% of students stated that podcasting changed their 
behaviors outside of the classroom when studying for exams. 

The integration of podcasting for instructional delivery also affected student’s study time allocation. About 
23% students reported that their study time decreased with the addition of podcast for content delivery. More than 
30% of students reported that their reading time decreased with addition of podcasts. Since listening or watching 
podcasts is individual learner-centered activity, more than 5 1 % of students agreed that learning via podcasting 
increased their study isolation and 31% of students reported that podcasting delivery decreased the involvement of 
community of learning. One of the suggestions made by students is that podcasting should be integrated with other 
learning activities. These data provide a very good guidance for the effective instructional design via podcasting. 

Podcasting and Student Learning Experiences 

Student enjoyment of the course is positively related to their learning attitudes and to their perceived value of the 
course they take (Patti & Saroja, 2005). Learning enjoyment also has been explored to design game-based 
instructional delivery to motivate and engage student learning. iPods and MMDs have been used for entertainment. 
The study data indicate that most students’ primary use of their portable devices is to listen to music. Will the 
enjoyable experiences transferable to their learning process? More than 53% {Figure 6) of students stated that 
podcast course delivery format made their learning more enjoyable. Approximately the same amount of students, 
52% {Figure 7) reported that podcast delivery format enhanced their learning experiences. The best types of 
engagement stem from the learner’s enjoyment of a more effective learning experience, one that puts them in control 
and encourages active participation, exploration, reflection and the individual construction of meaning (Galameau, 
2005). Podcasting is becoming a mutual communication tool instead of one-way delivery tool. Instructors can take 
advantage of this delivery media to possibly improve student learning effectiveness. 
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Figure 6(left) Podcasting made learning more enjoyable 
Figure 7 (right) Podcasting enhanced student learning experience 
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Podcasting and Student Class Attendance 

The most valuable features that students found with podcasting were flexibility and mobility. The data 
show that more than 83% of students favored with these two features because these features fit perfectly into their 
living condition and dynamic life style. University of Houston is a large urban commuter school. Majority of 
students have either full-time or part-time jobs. They spend a lot of time on the road driving to school and struggling 
to find a parking spot. This is especially difficult for morning class students. Our study indicated that the students 
who had morning class preferred podcasting to attending class lectures. When the morning class instructor started to 
deliver the full class content via podcasting, more than 85% of students accessed podcasting material instead of 
coming to class. For the other afternoon class, majority of students still attended class regularly. Besides, the 
afternoon class used podcasts as supplementary materials to lecture content. This is the reason that the following 
Figure 8 shows the percentage of students who preferred podcasting (40.52%) is almost the same as the students 
who preferred lectures (39.87%). More than 65% of students preferred the combination of podcast content delivery 
and face-to-face learning experiences. 

Figure 8 Student preference of podcasting to attending lecture class 
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Podcasting and Learning Effectiveness 

The ultimate goal of integrating podcasting in education is to provide a new instructional delivery channel 
to facilitate student learning and to improve their learning outcome. The quality course design is not determined by 
teaching or designing experts but by students, if they learn, how they learn and what they learn. Podcasting delivery 
quality directly affects student learning quality. The following data reflect student’s perception towards podcasting 
quality and their learning outcome. 
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Podcasting Design Quality 

Podcasting design quality was measured through content delivery accuracy and clarity, logical presentation, 
as well as the integration with other content and activities. In this study, with four classes, more than 73% of 
students reported that the podcast material content was clear and easy to understand. More than 71% of students 
agreed that the podcasting content was presented in logic order. More than 72% of students reported that the 
additional podcasting material was relevant to course content. More than 70% of students indicated that the podcasts 
were well-integrated with other class activities. Those class activities included discussion board posting, hands-on 
projects, group activities, and varieties of assessments. 

Podcasting and Leaning Effectiveness 

In this pilot. Podcasting was used for both full course content delivery and supplementary to lecture content 
to facilitate student learning. More than 66% of students reported that the addition of podcasting material was 
helpful to the understanding of course content. More than 59% of students indicated that listening to podcasts helped 
them better retain the content information. 

Last but not the least, when students were asked if the use of podcasts improved their grades, more than 26% of 
students reported positively. Figure 9 shows the grade comparison between the two fully online classes with same 
content material and taught by same instructor. Fall 2005 class did not use podcasts, while Spring 2006 class used 
podcast. There is a 12% increase with A students, and overall performance has been improved with the class using 
podcasting. The comments from students in the survey are also very positive in using podcasting for instructional 
delivery. 

Figure 9 Grade comparisons between two classes 




Podcasting Best Practice Generated from the Study 

Here are several recommendations based on the data collected from the research study regarding relating to effective 

integration of podcasting technology in instructional delivery. 

1 . Always have a back-up plan. Podcasting should not be the only delivery channel for instruction. You can 
implement it together with other delivery media, such as streaming, and CD/DVDs. Do not forget text, which is 
always the safest way to prevent any technical difficulties. The study results indicate that the students who had 
dial-up Internet connections had problems downloading large file size video podcasts; some students had 
limited storage space on their MP3 players for a full-length lecture podcast. Preparing material with different 
formats can satisfy different users and guarantee every student equal access to the course content. 

2. Re-structure the content, and “cuf ’ it to meaningful segments for effective delivery. Keep your podcast short, 
especially a video podcast (Vodcast). For an audio podcast, it should be under 30 minutes, for an enhanced 
podcast, under 20 minutes, and for Vodcast under 15 minutes. Several issues arise when using long podcasts. It 
takes longer a time to download the content; it takes up more space to store it, and some MP3 players have 
limited battery power. 
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3. Incorporate other learning activities with podcasting content delivery. There are varieties of learning activities 
that can help recall the information delivered via podcast, such as discussion forums, assessments, projects, etc. 
Faculty members can also create learner-centered activities to improve students’ ability to apply knowledge 
application. They may also provide students opportunities to re-construct the information for better 
understanding, and strive to find out the best teaching pattern favorable to student learning. 

4. Make the Vodcast complimentary to the information rather than replication of the information. Effective use of 
the video images can greatly increase the stimuli of the information to the learner’s brain. Instead of showing a 
“talking head” on the screen, extra visual information related to the content can enhance the learning effect by 
triggering new focus and attention. Do not use technology for the sake of technology. 

5. Use the lowest format to achieve best results. Here, lowest format means an audio only podcast. When you 
analyze the content, start with the lowest format of delivery. An audio Podcast has a smaller file size and takes 
less time to download. In addition, you can reach more students. The study results from the UH School of 
Communication show that more than 65% of students own iPods or IM3 players, but only one third of this 
population has an iPod with video capabilities. You should choose enhanced formats only when they can 
provide additional value to the content; otherwise, the audio Podcast is just as good as (or better than) the 
others. 
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Abstract: This study investigated the effects and challenges of a real-job project approach as pedagogical and 
curricular strategies in an instructional design and technology training program. Using questionnaires as a research 
tool, the study surveyed experiences and perceptions of professionals who graduated from the program in more than 
30 business and industry areas. The results show that trainees reported positive benefits through the real-job project 
approach, although the rate at which they had applied their projects in workplace was far below expectations. While 
this approach allows participants to initiate responsive solutions to their job settings, five elements (project vision, 
formative assessment, responsive activities, technology scaffolding, and workplace client) merit further 
consideration to help trainees develop more successful projects and exert greater impact on their workplace. 

Introduction 

Professionals are in a world of continuous change and innovation, and this is especially true for the field of 
instructional design and technology (IDT), where professionals need to utilize emerging technologies and design 
principles to provide new educational solutions. Training has been seen as a common means to keep today’s 
employees secure and competitive in a rapidly changing workplace. However, the traditional lecture-based training 
approaches have a low transferability of knowledge and skills to the trainees’ working place. Such perceptions are 
prevalent in the training field. Georgensen (1982) made an assertion that only 10% of the training investment has 
been translated into employees’ job performance. Shaw (1995, p.62) shares a similar observation, “As much as half 
of [the money American corporations annually spend on formal training] is being utterly wasted.” In addition to 
other reasons, such as training aimed at non-training problems, one major reason for the failure of training is its poor 
pedagogical design. Haskell (1998) claimed that the traditional, industry-based methods of training are inappropriate 
and unworkable for the fast changing, knowledge-based organizations. Today’s organizations must find ways to 
give their employees customized, thought-based, and task-extended training, instead of uniform, simple rote, and 
task-limited training. 

This study examined project-based learning incorporating real-job tasks in an instructional design and 
technology (IDT) training program. A real-job project approach is conceptualized as project-based learning 
incorporating trainees’ job components into their training process. The term “a real-job projecf’ is introduced to 
reflect the integration of authenticity in a project approach in the instructional context. This study investigated 
whether this real-job project approach could serve as an effective training strategy providing customization, 
relevance, usefulness, engagement, and application to the workplace, and how we can improve such a training 
design for future or similar IDT programs. 

Description of the Certificate Program 

The Instructional Design and Technology Certificate (IDT) Program was conceived to serve training 
professionals in a nearby metropolitan center as part of a college-wide outreach initiative. The goal of the IDT 
training program is to assist professionals in various organizations to better use the knowledge and skills of 
instructional design and technology in their working contexts. The training content is organized around the analysis, 
design, development, implementation and evaluation (ADDIE) paradigm for training development. The pedagogical 
framework for the IDT certificate program is built upon project-based learning and a job task from trainees’ 
workplace. This five-weekend training program, a total of twelve hours each weekend, is designed for professionals 
whose current or future job responsibilities involve designing training within an organization. The participants in 
the IDT program include trainers, managers, human resource practitioners, and other professionals responsible for 
training services. The training program is offered every spring and fall semesters and lasts five weekends, a total of 
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fifty-five instructional hours. Ten training sessions have been conducted to date and more than 100 people have 
participated in the program. There appears to be a perception of success. 

The training sessions are organized to support the project progress although no instructor is directly 
responsible for the project. Each session covers a different topic in the field of instructional design and technology. 

The topics covered are illustrated in Figure 1. 

Topics in the IDT certificate program 

1 . Introduction to the field and the theory of instructional system design 

2. Analysis: needs analysis, task analysis; trainee analysis; performance analysis 

3. Design: create the blueprint, storyboarding, tool choices 

4. Development: message design; development tools; studio experience 

5. Implementation: E-leaming issues, case studies 

6. Evaluation: usability testing; evaluation plan and tool 

7. Others: adult learning; project management; training proposal 

Figure 1. Substantive course topics. 

The trainees were introduced to the IDT training program and informed that the training is organized 
around a real world project by the program chair during the first week. They were invited to begin thinking about a 
software tool to learn, Authorware, Dreamweaver, Captivate, or Flash (the choice is subject to change from semester 
to semester). The rest of this week was to introduce trainees to the field of IDT, the ADDIE model, and 
instructional analysis. 

The focus of the second week was mostly about analysis including an analysis workshop, case studies of 
instructional design, adult learning theory, and project management. Trainees worked on their own training issues 
as class exercises, and heard real stories from industry and business. The adult learning session helped them be 
aware of adult learner’ characteristics and apply instructional principles for adult learners. The project management 
session helped them manage a typical project management plan. 

The third week was mainly about design and evaluation. The goal for this week was for materials to be 
produced on screen, like the color issue, font issue, and sequence issues, and trainees began to work on a storyboard 
for their ready-to-go product. Evaluation planning, tools, and usability testing were also covered during this week, 
although in trainees’ projects the evaluation part is never actually included. 

The fourth week mostly focused on development. Trainees worked in a computer lab, learned software 
tools, and developed online-prototypes. The class was broken up into groups based on the tools they had chosen to 
learn. Each group had an instructor to lead with a software book. They were located in the same computer lab, but 
generally trainees who worked on the same tool sat close to each other. From Friday to Saturday, instructors moved 
on to more advanced functions to help trainees develop their projects. Some student volunteers in the instructional 
technology department also came to help trainees with their tool learning. 

On the fifth weekend, trainees spent most of their time on their project work in the lab. A session about 
trends in the e-leaming industry and a session on training proposal writing were scheduled. On Saturday morning, 
project presentations were scheduled and each trainee presented on his or her own e-leaming product for around 15 
minutes. The presentation served as the culminating activity of the project. After that, a graduation luncheon was 
served and the certificate was presented. 



Design Characteristics of Pedagogy 

In order to help those professionals in various organizations effectively apply the knowledge and skills of 
instmctional design and technology, the considerations of the design of the program were given to blending 
authentic learning, project-based learning, and the nature of instmctional design. Specifically, the pedagogical 
characteristics underlying the training program are: 

a) The certificate program offers trainees the opportunity to choose an authentic project that is meaningful to 
trainees. The training is not a technical skills oriented program, but, rather, is intended to prepare trainees to 
effectively address and manage their training efforts. Trainees are encouraged to develop an authentic project 
related to their real-life working contexts so that they can apply the knowledge and skills they have learned. 
This characteristic allows the trainee to solve the real problem by hands-on doing. 

b) The training is organized following the framework of the ADDIE model. ADDIE is a generic model of 
instmctional design, which consists of five phases of Analysis, Design, Development, Implementation, and 
Evaluation. It is a well-known model for developing new training programs. Both the IDT program and the 
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project are built around this model to give trainees a sense of workflow in the community of practice. This 
characteristic makes sure that different sessions and learning tasks are organic connected and the framework 
lays out a core basis for the expected expertise. 

c) Instructors in the training program present theories from the academic world and stories from the real 
world as well. Another distinguishing characteristic of the IDT training program is the diversity of 
instructors. They include both faculty members from the university who conduct academic research and 
professionals in corporations who apply knowledge of instructional design and technology in the field. 
Including various speakers from the fields allows trainees to be familiar with IDT applications in fields and 
therefore, facilitates their own problem-solving skills. 

Methodology 

This study was designed to help us better understand the effects on participants’ experience from using the 
real-job project approach in the training context. It was also designed to provide ideas about how we can improve 
this approach in the context of training IDT professionals. Specifically, two questions were asked: What are the 
effects of a real-job project approach on trainees’ experience of learning instructional design and technology? What 
challenges do trainees encounter during the various stages (project initiation, project development, and subsequent 
workplace applications) of real-job project-based learning? 

This study employed online questionnaires as the primary method of data-collection. The questionnaire 
consists of both Likert-type and open-ended questions. The items in questionnaire include: choice of project; 
trainees’ entry levels in terms of computer expertise and instructional design expertise; three self-report values: 
appreciation of the project approach, achievement through the project, and application of the projects. The items 
also include different aspects of the project experience and different barriers trainees faced during the project. The 
open-ended questions ask about the reasons preventing further applications and their recommendations for 
improvement of the approach. The complete questionnaire could be found in Apprendix. 

The survey was conducted over a three-month period (February - April 2006). The researchers sent an e- 
mail to each participant to solicit their participation in the survey website. Quite a few participants had already 
changed their job and their contact information was no longer valid. Nearly 45 e-mails were returned due to invalid 
e-mail accounts. To ensure as large a sample as possible, participants with failed emails were then sent a letter by 
mail. However, still more than 20 letters were returned. Total responses were 43, representing 33% of total 
participants. At the end of the survey, participants were asked if they would like to participate in the interviews. 
Twenty-two participants chose to participate by entering their names and e-mail addresses. Upon completion of the 
survey, survey data were entered into SPSS software for descriptive analysis on each item and t-test on the 
difference between the real-job group and the hypothetical group. Open-ended questions were analyzed inductively, 
looking for themes within major categories. 



Results 

The sample population consists of 43 past trainees, representing one-third of the population across 12 fall 
and spring sessions from 2000 to 2005. These trainees represent more than 30 different various organizations, such 
as higher education institutions, government, public service (e.g. library), telecommunications industry, law agency, 
airlines, manufacturing, corporate consulting, independent contractor, etc. Approximately 65% (28) of survey 
participants come from the most recent three years. Sixty-five percent of participants chose a job-related project for 
their training, and 35% of participants chose to do a hypothetical project (Figure 2). The demographics of entry 
levels in terms of computer expertise and instructional design are shown in Figure 3. Participants rated themselves 
quite differently on their entry level of computer expertise (ranges from 1 to 10) and instructional design expertise 
(ranges from 1 to 10) on the 10-point scale. This spread-out pattern of entry levels might be an indication of diverse 
participants from various organizations and with different backgrounds. Trainees felt they were more skillful in 
technology than in instructional design. Most trainees reported a higher level expertise with computers compared to 
instructional design. In contrast, more than 60% of participants reported a lower level of instructional design. 
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Figure 2. The choice of project 
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Figure 3. Participants’ entry levels 



The results show that the real-job project approach in the IDT context is perceived as a fairly effective 
training model according to the self-report values of appreciation and achievement, although the full application of 
their projects seems far less than ideal. Most IDT participants perceived a fairly high overall- value of the project 
experience (Mean=7.53, SD=1.96), and reported a fairly high achievement value by doing projects (Mean=7.09, 
SD=2.22) on the 10-point scale. Such perceived values were also confirmed by both novice level and experienced 
professionals, 

'7 feel that the IDT certificate program as a great start to my career. I has [have] only been an ID for 3 
month, and this help me understand the overall concept. I appreciate the opportunity you gave us during 
this certification. ’’ [Novice professional] 

“I en]oyed the program immensely. Because I’ve been in the interactive industry so long — 15-20 years — 
much of what I learned on-the-]ob was not taught in formal classroom settings. It was good to learn the 
foundations and apply it to the details that I already knew and had been practicing for years. ’’ 
[Experienced professional] 

However, the application rate of their projects in the workplace is still far from the expectation 
(Mean=3.58, SD=3.07 on a 10 point scale). Only 3 participants in the sample population reported a full application 
of their projects. Nearly half the participants reported they were unable to use their projects developed in the class 
in their workplace at any level. While this approach allows participants to initiate responsive solutions to their job 
settings, both pedagogical and non-pedagogical mechanisms are needed to support a greater level of applications. 
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Figure 4. Self-report values on appreciation, achievement, and application. 



An examination of effects along the project process also indicates perceived benefits with this approach. 
During their projects, participants reported a couple of favorable experience with the choice of a project 
(Mean=4.61, SD=.83), motivation with the project (Mean=4.30, SD=.89), enjoyment of project development 
(Mean=4.07, SD=1.01), knowledge integration (Mean=4.10, SD=.76) on a 5-point scale. These values have overall 
small standard deviation, which means people generally have same agreement with these experiences. Especially, 
the choice to develop a job-related project was highly appreciated; more than 75% participants unanimously viewed 
the choice as a positive factor in the training program. As described by one participant, ‘7 en]oyed the IDT session 
and feel that having a pro]ect of our choice to work on is extremely helpful E While more than 70% students 
reported interactions with classmates were beneficial (Mean=3.98, SD=1.17), there is no significant evidence that 
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interactions with their workplace colleagues were extensive (Mean=2.51, SD =1.41); half of participants reported 
that interactions with their colleagues either did not exist or did not contributed to their projects. 

Trainees rated a little bit low on items of satisfaction with their final projects, usefulness of knowledge and 
skills, and impact on performance than other experience items, 3.70 (SD=1.12), 3.98 (SD=1.14), and 3.53 (SD 
=1.03) respectively. The standard deviation on these items also seems larger than other items, which indicates these 
experiences are quite case-dependent. The self-report values on different aspects of the project confirm earlier 
overall measurement that trainees generally valued the project approach, but perceived a little low level of 
application and connection between training and workplace. 

In terms of barriers participants encountered, participants viewed the insufficient time as a phenomenal 
barrier during their projects (Mean= 2.83, SD =1.41), and followed by lack of technical support (Mean=1.86, 
SD=1.07), and lack of support from the workplace (Mean=1.73, SD=1.00), and a clear project idea and direction, 
1.71 (SD= .97) and 1.69 (SD= .92) respectively. These barriers were also later confirmed by the open-ended 
questions and interview data. 



Barriers during the project 
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Lack of clear directions 
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1.69 
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Figure 5. Barriers with project development. 



Since participants chose to do either a real-job project or a hypothetical project, further investigation was 
conducted. T-test was used as an analysis tool to detect the difference between those who chose to do a real job 
project and those who chose to do a hypothetical project. Just as expected, trainees who chose to a job-related 
project have a significantly higher rate of applications after the training than trainees who did a hypothetical project 
(t(41)=2.52, p=0.16). Although the real-job group and the hypothetical group differed in their ratings on the 
appreciation and achievement, both are not statistically significant, t(41)=1.50, p=.14, and t(41)=.34 p=.74 

respectively. The difference on the item of appreciation seems educationally significant with one point difference 
on a IQ-point scale. 
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Real-job 


7.86 


7.18 


4.39 


1.90 


2.23 


3.11 


28 


Hypothetical 


6.93 


6.93 


2.07 


1.98 


2.28 


2.40 


15 


Total 


7.53 


7.09 


3.58 


1.96 


2.22 


3.07 


43 



Figure 6. Descriptive Statistics on Means and Standard Deviations for two groups 



For aspects of the project experience, two groups differed in the items of Q6a (t(18)=4.00, p=.00), Q6b 
(t(14)=2.29, p=.04), Q6h (t(41)=2.58, p=.01), Q6i (t(41)=2.22, p=.03), which indicates that the real-job group valued 
more the choice of the project, was more satisfied with their final products, and was better able to apply learned 
knowledge and skills. There is a small difference on all other items as well, although it is not statistically different. 
The evidence that the two groups differ illustrates that an authentic project raises trainees’ appreciation, application, 
and achievement to certain levels and aspects. T-test on barrier items indicates no significant difference, which 
means both groups had equal need for time and scaffolding. Additionally, both groups have no statistical difference 
in terms of their entry levels as well, which probably indicates whether choosing a job-related or hypothetical 
project is not correlated to their entry levels. 

According to the data, perceptions and suggestions for improvements have been categorized into five 
categories: 

1. Build the vision for projects at an early stage. For example, program organizers and instructors need to 
facilitate the plan of projects, including topic initiation, project process, and its connection to the IDT job. Some 
specific comments include: 

“Many people are indecisive or unaware of a good "problem " to use as a project. ’’ 

“Better balance between IDT concepts (the ADDIE model) and other factors of putting together a project - 

with emphasis on how to plan a project. ” 
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'7 think helping everyone understand that what they are doing in IDT is really cross functional project 
management and that they should have a good project management system in place before they even start 
needs analysis and the ADDIE process to be able to track all pieces of the project . '' 

'Did not enough explanation of processes to do project . '' 

"Explain the project and the goals of the project earlier on in the program so that people can start thinking 
about it and planning for it . '' 

"Need much more formal sessions for the project - many hit & miss . '' 

The strategies to build the vision include providing “more exemplary projects at the beginning” and 
contrast examples, “demo[nstrating] upfront of sample learning products made w/ each type of s/w,” and “allow[ing] 
past graduates to come back for a refresher.” 

To help trainees make the right choice with the development tools is also important. Many participants 
complained of the difficulty to match the tool and project. Some typical comments include: 

"Didn ’t know enough about the software to make an intelligent choice beforehand'' 

"I feel the program would be more beneficial if the students started their project the first or second week of 
the program. Give the students the chooses of what computer program for their project. If there is a 
website that they can go for trial use of each of the computer program, then they can let use know what 
program they want to use. " 

"Comparisons of common tools (Flash, Dreamweaver, Captivate, RoboHelp, etc.) used to develop 
instructional programs such as; skill levels, learning curves, advantages and disadvantages of each tool, 
integrating different tools to produce the final result, etc. " 

2. Define project structure and integrate formative assessment at every milestone. Besides a good vision 
and a right choice with tools, participants need to have a project line to flow with and check points are necessary to 
make sure quality of the project. The whole project could be divided into organically related sub-projects and the 
output of the early stage will be part of the input of the later stage. 

"It seems to be a little difficult to draw connections of the content [presentations] between each. Would be 
helpful to have an outline or some objectives to tie everything together. " 

"more small, iterative steps (mini-projects) building to final project. " 

"I am still a little unclear of any intermediate deliverable surrounding the final project. " 

Meanwhile, participants need formative assessment and feedback for their intermediate deliverables. 
Participants felt that there was nobody to critique their assignments and, therefore, were not sure if they were on the 
right track. 

"A CHECK Point along the way " 

"More time for review with instructors. During class we were developing our project and had clear 
instructions with a deadline but when we came to class with our completed assignments, the instructor did 
not go over it with us. " 

"Require documentation throughout the whole process. All approved projects in a put in a pipeline and 
managed through the project management shop. " 

IDT projects in the program were actually followed with the ADDIE framework, and inter-mediate 
deliverables were the documentation for each stage. ADDIE deliverables were implicitly a part of instructional 
design but were not explicitly required and assessed for the project, which made the trainees, especially the novices, 
confused their project journey. Providing procedural scaffolding and giving feedback for intermediate deliverables 
will guarantee a more successful project. 

3. Gear presentations and activities toward trainees’ projects and potential applications. Professionals from 
business fields introduced various end products and practices, which helped trainees widen their horizons. 
However, from the trainees’ perspective, they would like to know more about the process of design and 
development, rather than the end product itself As participants commented: 

"All speakers were very knowledgeable; however, they need to adjust their presentations to how each 
subject applies to the students and/or our jobs. " 

"Recommend speakers talking less about their business and more about how we can apply skills. " 

"Some of the business professionals focused more on their product and not how they erected the product. " 
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Trainees also would like to have more hands on activities which are related to or inform their projects in 
each session. 

“I would like more hands on exercise . '' 

'‘As they go through the IDT program with each lecture, use part of the weekend to practice what they 
learned in lecture as it relates to their project(for example-project management, storyboarding ). '' 

“As a class we needed more opportunity to participate and discuss how what we learn can be applied back 
on the job . '' 

“More instruction on storyboarding . '' 

Given the fact that many speakers presented in the training, more coordination in term of the trainees’ 
project, rather than just the content itself, would make the projects a more collaborative effort. The individual 
instructor for each class could include activities and discussions responsive to the trainees’ projects. 

4. Overcome time constraints and provide technology scaffolding. Having insufficient time for their 
projects is a major challenge for participants, mostly because of the challenge to master computer tools. As 
participants commented, 

“Not enough time learning software. ” 

“I would have like more computer time to work on my project & get assistance before final weekend. ” 
'Would even be willing to come an extra weekend to learn specific computer program. " 

“While I was lucky and had time outside of class to work on my project, I am sure that there were others 
whose work environments were not as flexible with their time. Therefore, perhaps a couple more hours of 
classroom time for project completion could be beneficial for some. " 

More technology scaffolding is seen as a major way to facilitate the development process. 

“I felt that more lab time was needed to complete the project. It may have helped to have more 
training/teaching assistants in the lab while working on the projects. I had never used Flash or 
Dreamweaver before and I was trying to learn the software while designing a program. It was a little 
difficult but I managed. I would have liked more time to learn the software and more time to work on the 
project with more assistants in the room. " 

“would be to have some extra support for less self -motivated and/or technically skilled students. Learning 
new software and creating a project with it was a welcome challenge for me, but some individuals panic at 
the idea of creating a project using software that they are unfamiliar with. " 

However, scaffolding is not just a simply help. It needs to be a learning opportunity, and needs to be 
cognitive promoting and challenging. Just fixing a problem for trainees does not help their expertise in long term. 
As explained by one participant, 

“I didn’t feel that we learned Flash but that the students helping in the lab simply fixed any ’’’’bugs”” for 
us. " 



Along with scaffolding, the time constraints could be also overcome by adjusting the lab schedule or 
making use of a group project. Some specific comments include: 

“More time allocated, perhaps earlier in the schedule. " 

“Get to the lab sooner! Also, I was very impressed with how my colleagues used the software, but there 
was no time to get coaching from them. Some of them were much more skilled than I at figuring out what 
the software would do, and using it to best advantage. " 

“It might be good to work on a project as a group. You don’t get enough technical training to really be 
savvy in creating slick IDT projects - - and that is what the business is looking for. " 

5. Explore an explicit mechanism for interactions between the training and workplace. Some participants 
commented on the need to strengthen the connection between training and workplace. As mentioned by a 
participant, “the more you can tie the class into the learner's work environment, the better.” Inviting a project client 
from the trainee’s workplace might increase involvement and interaction and, as a result, foster a more responsive 
project. 

“I would have the participant select a sponsor at the workplace to act as a catalyst to the project during 
the certification and an advocate for the employee as they take on additional tasks at the workplace. " 

“And there is also a need to understand prioritization of projects and to communicate the prioritization 
process to clients to set the right expectation for delivery. " 
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Although few participants proposed inviting a client for their project, this suggestion seems valid given the 
fact according to Q6d that there are only limited workplace interactions and involvements. The client could be 
asked to review the trainee’ documents and/or sign the approval documentation. A client model paired with the real- 
job project approach might facilitate a deeper level application. 

Conclusion 

From the trainees’ point of view, the real-job project approach in the IDT training is welcome because it 
provides “opportunities to produce end products and apply skills” and allows students to “have a goal to 
accomplish.” Data substantiated that the trainees developed appreciation and competence through the approach, and 
perceived it appropriate for working adults, although the application of projects was not at ideal level. The approach 
does not bring in both intensive and extensive applications as expected. More detailed research revealed that the 
major reasons why those projects could not be applied in the workplace include the nature of the project, 
individual’s job change, workplace constraints, pedagogical constraints, and technological constraints. While not all 
these issues could be addressed instructionally, the training could at least, through a design perspective, aid trainees 
to determine the project’s direction, scope, and size, and foster more interactions between workplace and training. 

Comparison between the real-job project group and the hypothetical project group favors the real-job 
project group in terms of appreciation, achievement, and application, and different aspects of project experience. Of 
course, this study employed self-report measures, so the caution is necessary because using perceived appreciation, 
achievement, and application has its limitations. As Thomas (1999) pointed out, self-report measures are not 
measures of what happened, but what participants believe happened, and thus reliance on these measures can be 
deceiving. Triangulation was used to overcome this limitation. Triangulation of different data, as well as 
performance tests, and different sessions proves the data to be representative and valid. 

Caution is also necessary to generalize the group differences in this study, since the research design is not 
experimental. That is, real-job and hypothetical projects were not randomly assigned. Both groups were actually 
under the same treatment. The difference could result from other factors, such as participants’ disposition, 
personality, etc.. However, the fact that there is no significant difference in terms of entry levels seems a favorable 
condition in this study. Further research is needed to compare the real-job approach and the hypothetical approach. 

According to participants’ open-ended questions and interviews, whether to choose a job-related project 
also depends on their personal status, work progress, and job background. It is unfortunate that these factors are 
difficult to address from an educational perspective. However, more visioning and scaffolding at an early stage 
could facilitate the participants to target an IDT project for the workplace. More mechanisms for interactions 
between the workplace and training could also keep them on job-related projects and, therefore, facilitate 
applications later in the workplace. 

Blumenfeld et al. (1991) pointed out that project-based learning does not itself guarantee effectiveness in 
the K-12 context. Their view also fits in the training context. The project approach presents challenges for trainees, 
instructors, and institutions, and it necessitates a careful design to promote understanding, motivation, competence, 
and application. Five themes that require more pedagogy considerations in this study are building project vision, 
providing structure and formative assessment, gearing responsive sub-activities to project, overcoming time 
constraints because of tool learning, and facilitating interactions between training and workplace. These themes 
basically parallel what Barron et al. (1998) proposed as strategies in the K12 context (appropriate goals, 
scaffoldings, opportunities for formative assessment and revision, and social structure promoting participation). 
Given the context of instructional design and this real job approach, elements worth considering are IDT project 
vision, ADDIE documentation check points, tool scaffolding, and workplace clients. Consideration of these 
elements could facilitate more successful projects, more intensive and extensive applications, and deeper 
connections between training and workplace. 
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Abstract 

The purpose of this study is to provide insight into the reform efforts in Turkey that deem integration of 
computers into teaching to be an essential element. While issues related to computer infrastructure and teachers’ 
computer literacy are typically accepted as the most crucial obstacles in the implementation of computer-based 
technologies in Turkish schools, there may be other less visible, yet equally important, barriers. Using discourse 
analysis, this study identifies yet another factor that might decelerate successful computer implementation: 
teachers’ teaching philosophies. A focus group interview with seven Turkish teachers reveals that the 
‘transmission model of teaching’ is the main, if not the only, teaching philosophy these teachers operate under, 
the monopoly of which may set hurdles for both reform efforts and successful computer implementation. 

Introduction and Background 

In Turkey, as in most other countries, the integration of computers into the curriculum is generally 
associated with reforming education. Certainly, the availability of computers in schools and teachers’ adequate 
computer literacy level are essential conditions for teachers to incorporate this technology into their teaching. 
Accordingly, the Turkish Ministry of National Education (MNE) has invested significant amount of money in 
computers for public schools since 1984. These efforts gained a new momentum especially in the last 15 years. 
In order to provide models of computer use in a reformed educational system, new types of schools that were 
equipped with computers and new curriculum agendas were established, and teachers have been provided 
computer training to increase their competency with computers. While these improvements are necessary 
conditions to integrate computers into teaching, the low level of computer utilization of seven Turkish teachers 
in their teaching in a context where computers are available and these teachers consider themselves as competent 
computer users suggest the presence of other barriers. In this study, I explore one of these less visible barriers 
that is evident in the way teachers talk about their teaching with computers. My study shows that these teachers 
mainly operate under a ‘transmission model of teaching,’ the primacy of which may get in the way of reform and 
successful computer implementation. By identifying such a factor, I aim to draw attention to its implications not 
only for teacher education and teacher professional development programs, but also for the larger context of the 
educational system. 

Having realized that it had an insufficient computer-literate work force for the foundations of an 
information-based economy (Yedekcioglu, 1996, January), Turkey started to invest in computers in schools in 
1984 (Odabasi, 1998). In the following years, this enterprise has gradually coupled with reform ideas. In 
especially two prominent reform projects, it is apparent that computers have been considered as an essential 
input of the educational reform movement in Turkey. 

Milli Egitimi Geli§tirme Projesi (MEGP) (the Project for Improving National Education) is one of these 
projects that was initiated by MNE in 1990 in order to keep up with the change and development in 
contemporary education. ^ Two categories of schools have been determined as the practice areas of this project: 
Curricula Laboratory Schools and Computer Experimental Schools. These two types of schools have been the 
sites for pilot studies for the programs or projects developed by the Department of Education Research and 
Development (BARGED). These schools are equipped with the relevant instructional materials, the priority 
being computers and educational software, in order to apply those programs. Later, MNE started the Temel 
Egitim Projesi (TEP) (Basic Education Project) in 1998. This project has had three-fold goals: improving 1) 
physical infrastructure (e.g., construction of new school buildings or restoration of available school buildings), 2) 



^ This project is completed in 1999 (MEB, 2005). 
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educational quality (e.g., building computer labs), and 3) quality of teachers (e.g., teacher professional 
development for computer literacy) (MEB, 2005). ^ 

In both projects, one of the main goals has been to disseminate and establish computer aided education 
along with improving physical infrastructure of schooling. More importantly, both projects aimed the use of 
computers to stimulate reforming Turkish education. As one can see in the Basic Education Project, computer 
integration related goals are considered as a quality rather than a physical infrastructure issue. Furthermore, 
being able to use computers is considered as an essential characteristic of the ideal student these projects aim to 
cultivate. In Curricula Laboratory School model, an ideal student: 

[H]as scientific and rational thinking skills; is analytical and inquiring; is one who does not 
memorize facts [or knowledge] but one who can access it, use such knowledge, and able to 
share it; has communicative skills; is an effective user of technology [emphasis added]; fulfills 
one self and also possesses common shared human values; is creative, productive, cooperative; 
is one who knows how to learn and values life-long learning skills (LARGED, 2006, Ogrenci 
Merkezli Egitim, para. 1). 

Efforts aimed to solve infrastructure issues gained a new momentum in 2003. The government initiated 
the campaign ‘100% Support for Education’ in September 2003 that allowed tax-free donations for the national 
education. In this project, schools report their needs through the project website and philanthropists may aid a 
school according to these needs. It appeared that computer-related needs constituted the most significant portion 
of these reports and funds have been allocated toward this direction. While at the end of the 1998, there was one 
computer available for three schools (Ozden & Cagiltay, 2000), with this campaign, the ratio became one 
computer for 55 students. In the near future, as the campaign continues, it is expected that the ratio will be one 
computer for 35-40 students (Egitimde teknoloji devrimi, Aralik 2005 - Ocak-Subat 2006). 

Infrastructure related initiatives were also joined with support for teachers in these technologically- 
enriched educational environments. One of the support attempts for teachers who work in either Curricula 
Laboratory Schools or Computer Experimental Schools has been continuous teacher in-service computer 
training. In 1996, Yedekcioglu (1996, January) reported that approximately 100,000 teachers have taken in- 
service computer training. More recently, MNE collaborated first with Intel in 2003 and later with Microsoft in 
2005 in order to initiate teacher professional developments that were aimed to make teachers computer literate. 
To date approximately 80,000 teachers (about 13 % of the whole teacher population) received their certificates 
from these programs (Egitimde teknoloji devrimi, Aralik 2005 - Ocak-Subat 2006). 

The assumption underlying all these efforts has been that if schools are equipped with computers and 
teachers can become competent users, then computers will be used in education to realize the ideas of reform. 
This idea finds support from the literature on two levels. First, according to Stevens (1982) two main factors 
determine successful implementation of computers: teachers’ attitudes toward computers and their levels of 
expertise with computers. Studies have also shown the correlation between the two; that is, teachers develop 
positive attitudes toward computers as their competency level with computers increases (Madsen & Sebastiani, 
1987; Yildirim, 2000). Secondly, research suggests that the use of computers in classroom can promote the 
ideas of reform, such as student-centered instruction. In one of the most comprehensive longitudinal study of 
technology-using teachers program, namely Apple Classrooms of Tomorrow, Dwyer, Ringstaff, and Sandoltz 
(1991) found that changes brought by teaching with technology caused teachers to interrogate their beliefs about 
learning and efficacy of their instructional activities. The teachers in this study shifted from curriculum-centered 
towards student-centered instruction, from individual towards collaborative tasks, and from passive towards 
active learning (as cited in Topper, 1998). 

Given all these investments, the vision, and promises of using computers in education, it becomes even 
more important to recognize the obstacles that might decelerate successful computer implementation. Although 
infrastructure and computer competence are important factors for integrating computers into teaching, they may 
not assure that the intended goals of reform will be achieved. This study tries to bring light to one of the less 
mentioned and less visible barriers in the implementation of reform with computers in Turkish schools. By 
employing discourse analysis as the method, I examine how teachers talk about computers as an instructional 



^ The first phase of this project completed in 2003 and the second phase was initiated in 2002 (MEB, 2005). 
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tool. By doing that I aim to reveal the underlying teaching philosophies teachers hold. I argue that teachers hold 
a ‘transmission model of teaching’ and, coupled with their teacher-centered teaching practices, this view gets in 
the way of realizing the potential of computers for different teaching practices. For computers to play a crucial 
role in the reform movement, the monopoly of these practices and philosophies should change while changing 
technologies in Turkish schools. 

Below I draw from Case (1996) and Sfard (1998) to discuss different learning theories that are founded 
on different philosophical assumptions. I define the ‘transmission model of teaching’ based on the behaviorist 
learning theory, which is derived from British empiricism (Case, 1996) and rooted in objectivism (Jonassen, 
1991). Next, I use Case’s and Sfard’ s ideas to present alternative paradigms for knowledge and learning, which 
point the way to alternative pedagogies. Following Sfard and Boaler (2002), I emphasize that none of these 
views are superior to each other. Rather, it is the diversity of different views of knowledge from which any 
education reform can benefit. 



Philosophical Foundations of Learning Theories 

The roots of the first view of knowledge that gave rise to the didactic theories of learning lie in British 
empiricism (Case, 1996). In the field of education, the major and most influential theory that originates from 
empiricist philosophy has come to be known as behaviorism. Behaviorists define learning as the observable 
change in behavior, excluding the role of mind in learning from their analyses. Mind is viewed as a ‘tabula 
rasa,’ an empty vessel, waiting to be filled with the knowledge of the natural world. In behavioral (and most of 
cognitive) psychology, such knowledge is considered to be objective - out there - and transmittable (Jonassen, 
1991). Jonassen describes the assumptions of an objectivist approach to learning: 

Objectivism — the more common scientific conception of reality — holds that 
there is an objective reality that we as learners assimilate. The role of the 
education is to help students learn about the real world. Students are not 
encouraged to make their own interpretations of what they perceive; it is the 
role of the teacher or the instruction to interpret events for them. Learners 
are told about the world and are expected to replicate its content and 
structure in their thinking (p. 10). 

Thus behaviorist pedagogy typically regards learners as the passive recipients and underlines an 
instructional approach that focuses on the quality of the “transmission of knowledge” (Jonassen, 1991). From 
now on, I will refer such philosophy of teaching founded on behaviorism and objectivism simply as the 
‘transmission model of teaching.’ 

Although behaviorist epistemology has been the dominant paradigm for the conceptions of learning for 
most of the first half of the 19^^ century (Jonassen, 1991), empiricist and objectivist traditions have not had the 
sole impact on conceptions of knowledge, learning, and pedagogy. Rationalist and sociohistoric stances 
provided alternative and foundationally different directions in education (Case, 1996). In fact, much of 
educational reform has been advocated through these two paradigms. 

The rationalist view gave rise to the well known constructivist learning theory that has mainly 
originated from the work of Jean Piaget. Piaget’s theory states that mind applies existing knowledge structures 
to the new experience, and learners engage in an active process of making sense of the environment, and this 
process in turn determines the rate of cognitive development. Although constructivist learning theory evolved 
into different forms in time, such as radical constructivism and social constructivism (Ernest, 1994), its basic 
pedagogical implications remained with us: rather than treating the child as an absorber of transmittable 
knowledge, teachers should actively engage youngsters to build their own intellectual structures. Thus, while 
behaviorist pedagogical model emphasizes the one-way transmission of concepts from teacher to the students, 
constructivism is based on the belief that knowledge emerges from interaction. 

For the last two decades, however, educational research has been influenced by another paradigm shift. 
While Case (1996) names the new perspective sociohistoric, Sfard (1998) examines the resulting change in 
learning theories in terms of metaphors. In recent publications, Sfard detects the emergence of a new metaphor 
of learning, namely the participation metaphor (PM), which underlines a different view of knowing. This view 
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lies in contrast to the acquisition metaphor (AM) that regards knowledge as substance, and learning as the 
acquisition or construction of that knowledge. Thus, Sfard considers both the behaviorist and constructivist 
theories of learning under AM. PM, on the other hand, views knowledge as the aspect of 
practice/discourse/activity, and learning as becoming a participant in a social context. Theories of situated 
cognition that put an emphasis on learning as a legitimate peripheral participation (Lave & Wenger, 1991) are 
characterized under PM. Sfard holds that in PM “the permanence of having [emphasis in original] gives way to 
the constant flux of doing [emphasis in original]” (p. 6). Learners are characterized as being interested in 
participating in certain kinds of activities rather than being in the position to accumulate private possessions. 

However, it would be too simplistic to decry AM over PM, as Sfard (1998) wisely reminds us. These 
two perspectives perhaps complement each other in a way analogous to the Tight’ phenomena in physics and 
illuminate and guide classroom practices in appropriate moments (D. W. Shaffer, personal communication, April 
2004). Moreover, the dominance of a single theory in educational research and practice has been criticized on 
the grounds that it inevitably narrows our thinking and limits progress in any educational innovation (Boaler, 
2002). Thus, it is the multiplicity of different views of knowledge from which any educational innovation can 
benefit. In fact, in the U.S., we see that the ideas of mathematics education reform promoted by the National 
Council of Teachers of Mathematics ([NCTM], 2000) are consistent with more than one approach, both 
constructivism and the psychological theory of situated cognition (Senk & Thompson, 2003). 

Methodological Framework: Discourse Analysis 

The methodological tool employed in this study is discourse analysis as conceptualized by Gee (1999). 
As discourse analysis is the analysis of language in context, the tools of inquiry it provides help to illuminate the 
aspects of the context where the conversation takes place. Gee argues that language has a magical property of 
both reflecting and shaping the context. From that perspective, meaning is always a function of ‘now’ and 
‘before.’ Gee defines two interconnected primary tools of inquiry for a discourse analysis that address now and 
before: ‘situated meanings’ and ‘cultural models.’ A cultural model is what we bring to the situation, our taken 
for granted assumptions about what are normal or typical in order to both function and understand the context 
where meaning is situated. A situated meaning, on the other hand, is what something means at that particular 
context connected to the cultural models it is attached to. This mutual way of reflecting and constructing the 
social world in turn gives meaning to the situated identities the participants enact and situated activities they are 
involved in a particular context. Gee defines the language (social language) used to construct these situated 
identities and situated activities as discourse. He further argues that it is not simply the language that is 
important. We also use certain ways of speaking, being, acting, thinking, valuing, believing, knowing, and so 
forth, to be recognized by others as particular ‘who’s’ engaged in particular types of activities. Thus the ‘big D’ 
Discourse comprises both language and non-language ‘stuff that we use to display and recognize specific 
situated identities and activities. 

Based on this framework, I consider teachers’ theories about teaching or what they take for granted as 
the role of the teacher as a ‘cultural model’ that reveals itself in the way they talk about computer use in their 
teaching. This framework also enables me to pay attention to the context of data collection in which I and my 
participants enact situated identities within a situated activity. 

Method 

The data that I analyze in this paper is a focus group interview that I conducted with seven Turkish 
teachers. These teachers work at the same inner city Ilkogretim School, which is located in an affluent 
neighborhood of Istanbul, Turkey. An Ilkogretim school is the combination of five-year elementary and three- 
year middle school, which has become compulsory since 1997. This school is a Curricula Laboratory School; 
that is, the school is equipped with a computer lab and educational software, and these teachers have had several 
in-service computer training courses. They also define themselves as “experienced” in their computer 
competence. Of the seven teachers, three of them are elementary school teachers, two are mathematics teachers, 
one is a computer teacher, and one is an English teacher. The average teacher age in the group is about 42 with 
an average of 22 years of experience. 

During the interview, I asked three main questions: 1) how do you evaluate your teaching with 
computers? 2) what do the individuals or groups of individuals, who are important for you or whose opinions 
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you value, think about your use of computer-related technology in your classroom? and 3) considering the 
factors or circumstances that may facilitate or impede your teaching with computers, how much control do you 
believe you have over using computers in your classroom? The interview was audio-taped and transcribed. I 
translated the quotes from Turkish to English. 

Instead of numbering every sentence, I gave each turn a person takes a line number in order to keep the 
meaning in context. The smallest object of analysis is these individual turns that teachers take during the 
conversation. Most of the time teachers addressed or responded to one single issue when they took a turn. For 
this reason, I will call these turns “stanzas,” since they can be regarded as devoted to a single topic (Gee, 1999). 
Also when I give parts from the transcript, I first write the line number of the stanza and then speakers’ initials. I 
underlined the words that are stressed. I excluded some irrelevant sentences and used bracketed ellipses to 
indicate that. Ellipses at the end of the sentence shows that the sentence is not finished. The statements in 
brackets are either my or other teachers’ comments or reactions to what has been said, or the possible words or 
phrases that I think the speaker intended to say. The stanzas in italics are my own. 

In analyzing data I first categorized the whole interview into themes/motifs, the parts of interview that 
correlate with each other. In this way, one can set the analysis in the larger context of the whole interview (Gee, 
1999). Then I used these to form a hypothesis about some of the cultural models. Gee says one way to get at 
people’s cultural models is to ask “what I must assume this person (consciously or unconsciously) believes in 
order to make deep sense of what they are saying.” After stating my hypothesis that I have drawn from the 
overall interview, I present the evidence for this from the parts of the interview where teachers have talked about 
the way they employ computers in their classes. Then I look for coverage and convergence (Gee, 1999), that is, 
whether the hypothesis I have generated illuminated other parts of the interview and whether the ideas from the 
other parts continued to support it. My analysis has also included the activity and identity building of my self as 
well as that of the teachers to sustain my argument. 



Results 

Looking at the whole interview, I identified three main issues teachers addressed, which are relevant for 
the present analysis: 1) their evaluation of the use of computers in education (regardless of whether they use 
them in the classroom or not), 2) factors that impede or help their teaching with computers, and 3) the way they 
employ computers in their classes or the way they think they could use them. Then I identified themes/motifs 
within these interview sections. In this way, I partitioned the interview into underlying ideas, and this helped me 
to form a hypothesis about the cultural models of teaching. I provided parts of the discussion when relevant for 
the themes. These themes and my organization of the interview into different topics are not necessarily mutually 
exclusive. Sometimes two or more themes can be found in a thread of discussion or some sections of the 
interview may speak to more than one issue I outlined above. Below I start exploring these themes both to 
provide the background and to build a hypothesis. Then I analyze sections of the interview to provide evidence 
for the hypothesis. 

1. Teachers^ Evaluation of the Use of Computers in Education 

Theme: Positive attitudes toward teaching with computers. All seven teachers, except one who is self- 
taught, learned to use computers in in-service computer trainings, and regarded themselves ‘experienced’ with 
computers. However, of seven teachers in this group two of them never taught with computers. And the others 
also reported that they are not using computers at a desired level because of the factors impeding their use of 
computers in class. In evaluating the use of computers in education, however, nearly all the teachers expressed 
positive attitudes towards using computers in education and in their classes. The majority of them said 
computers were very beneficial in education. 

Among the benefits of computers teachers mentioned, visual aspect, novelty effect, and saving time 
were the main ones. Teachers mentioned that computers can afford learning by experience and, given that 
students are fond of technology they would be more motivated to learn with computers. In addition, computers 
can help teachers to save time, since once they prepare their classes with computers, they would not need to 
write things down on the blackboard. One teacher said: 
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41. SI: If I used it how would I evaluate it? [repeating the question being asked] At least the kid by 
seeing, I draw these [the pictures] on the board, I draw the pictures on the board. It is a waste of time. 
But [if] I have the computer in front of me [. . .] if we show them [pictures] on the screen using the 
projector, then kids have fun w hile learning, I mean they learn by implementation . First of all we save 
time, we hit the button, and students see the picture right away and I just explain it. If students at least 
learn by seeing, hearing, experiencing. . . 

42. DO: Again you think it is entertaining. 

43. SI: Absolutely , absolutely. 

Theme: Hesitation and uncertainty about the real value of computers in education. Despite their 
positive attitudes toward teaching computers and locating computers’ benefits in teaching, two of the teachers 
first hesitated to make evaluations about teaching with computers. However, these statements were not 
completely opposing their positive evaluations, rather they were additional. One of the teachers said: 

24. AA: Since we do not teach with computers, to say something about it, [I: I hear you] it is kind of 
awkward [another teacher: yeah it is]. But I personally would like to teach with computers, I would 
want that, though I am not quite sure about what the actual result would be [I: mm hm]. I prepare my 
exam questions and the answer key on the computer, and I like that. But I do not know what the result 
would be [if I teach with computers]. 

25. DO: But what do you think about teaching with computers? 

26. AA: I say it is good [I: good], very good [I: very good]. It would be very beneficial , I think [I: you 
say beneficial] yes. 

The other teacher also expressed uncertainty about the real value of computers in education. This 
teacher made clear that she was uncertain about what computers can offer instructionally, which was due to her 
lack of teaching experience with computers (which was due to the impeding factors to teach with computers). 
She said: 



47. GA: Let me put it this way, science classes, especially science classes , umm. . . should remove 
blackboard and chalk. And as my friend articulated [learning] by doing and experiencing. . . 

48. DO: Do you think computers can provide that? 

49. GA: The computer. . . that’s I don’t know yet. Also the student. . . it [education] should be student 
centered, and the student should be questioning, thinking, and interpreting. However, I do not know yet 
if this can be done with computers , yes. Yes, it is good in terms of time management, and I agree with 
my friends with that [. . .] [but] there is only one small screen in our classes. Screen. . .1 mean the most. . . 

One other teacher, DG, also revealed some concerns. What was underlying his statements was the idea 
that computers might threaten teachers’ position as the center of knowledge. Below is the part of the interview he 
expressed these concerns. 



93. AA: (To DG) [. . .] could you tell n^ that ‘you my fellow teacher [hocam] if you do your math class 
with computers it would be better than the current situation?’ [. . .] your advice for me , ‘you my fellow 
teacher, if you teach like this with computers, by using this technology, you would teach more 
productively, more effectively, more productively?’ 

94. DG: There is this thing, you sit at your table and there is a big screen, and students look at that. 
“Okay children look at this now, here is the formula for that, here is the example,” and some things are 
going on on the screen. There is no such thing. If there were then why would you need a teacher? The 
teacher should own the knowledge at the end. The human factor should be available there. There are 
great computer programs. By just spending three hours on the computer daily, you can have two 
university diplomas. But the human factor here is important. 

95. AA: But that preparation is yours, you make that, you show it to students and then explain it on the 
blackboard. 
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96. DG: But we should be thinking about this, okay we use computers, for instance what am I going to 
show to students on the computer and what not to [AA: of course] I do not think that blackboard and 
chalk is that bad, I am telling you. 

2, Factors that Help or Impede Teachers ^ Teaching with Computers 

Upon being asked, teachers briefly mentioned factors that had been helpful for them to teach with 
computers. These were mainly teacher professional development trainings, parents’ and students’ expectations, 
and support from previous school principles. However, the majority of the time, they talked about factors that 
impede their teaching with computers, and they ascribed their low level of computer use to these factors. Two 
main themes emerged in this dimension. 

Theme: External obstacles. Teachers mentioned factors such as lack of infrastructure and lack of 
continuous support for the available infrastructure, lack of time to prepare and use computers in the class, lack of 
software, and lack of support from the school administration as obstacles to teach with computers. I categorized 
these factors as micro-level constraints as they relate to local structures. In addition, teachers pointed out the 
general problems of educational system in Turkey, such as, the system’s general emphasis on memorization of 
facts, low share of educational expenditures from the budget, low investment in computers, and finally 
corruption. These factors are categorized as macro-level constraints as they imply larger scale perceived 
obstacles. All together I considered these two types of factors as external factors, factors teachers think they 
have little control over. 

Theme: Internal obstacles. However, teachers also made statements that revealed their lack of interest 
and motivation for using computers. One teacher said: “But to me with what we are provided so far, something 
could have been done if [we] wanted. But. ..There is knowledge but... ’’ I considered these types of obstacles as 
internal, given that these were the types of obstacles teachers associated with themselves at the personal level. 
However, mention of internal obstacles was a lot less than the external ones. 

3, The Way Teachers Employ Computers in their Classes or the Way They Think They Could Use Them 

Theme: I [could] use computers to present the topic or after the topic is presented. The interview data 
also speak to teachers’ current practices with computers or how they could use them potentially. Here the use of 
computers to present the topic or after the topic is presented appeared as an underlying dominant theme. 

Teachers make use of computers (or they think they could use them) as the containers of their lectures or as the 
container of practice problems (as available in educational CD’s). Here we see that the main benefits of using 
computers are to save time in order to do more exercise problems, to motivate kids, and to get feedback. Below 
are two threads of discussion where this theme is evident. 

75. AA: [. . .] I am saying that if I prepared my lesson on the computer , on a diskette, did my practice 
and present the topic with the computer, I would be more effective, give more knowledge, and save 
more time. 

76. GA: By all means, we believe that. 

77. LK: There would be more time to do more exercise problems. 

78. AA: I would do more practice problems and make students comprehend the topic better. 

79. SI: And also [kids] learn by seeing. [Kids] find it more entertaining. 

80. AA: In short, if I teach with computers , the situation would be better than the blackboard and chalk 
environment. 

81. RA: [it] removes [blackboard] yes. 

82. LK: Also there might be things even we forget [to mention], we are also human. There [in 
computer] one can go back, immediately realize, realize things that we skipped. 



56. DO: What do you think Mrs. ... [to LK]? Do you think it is beneficial to you in your classes? 

57. LK: It would be beneficial, to if we were connected to the Internet. Then more. . . 

58. DO: I guess you have a computer lab with the Internet connection downstairs. 

59. LK: Yes but the lab had not been open [GA: the classrooms do not have it (the Internet)]. Last year 
its door was closed [locked]. 
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60. DG: The lab has the Internet [connection] but not the classrooms. 

61. LK: Not in classes. The lab was not open to us either. And for that wq could not utilize it. But I 
make use of CD’s. I insert the CD and, there with students, well in math there are practice problems 
usually. After the topic is presented, we enter those practice problems. 

Similarly, below SI suggests that the computer could make it a lot easier to provide practice situations 
for her students. This conversation takes place in the context where DG explains his concerns about computers, 
which I quoted earlier. 

94. DG: There is this thing, you sit at your table and there is a big screen, and students look at that. 
“Okay children look at this now, here is the formula for that, here is the example,” and some things are 
going on on the screen. There is no such thing. If there were then why would you need a teacher? The 
teacher should own the knowledge at the end. The human factor should be available there. There are 
great computer programs. By just spending three hours on the computer daily, you can have two 
university diplomas. But the human factor here is important. 

95. AA: But that preparation is yours, you make that, you show it to students and then explain it on the 
blackboard. 

96. DG: But we should be thinking about this, okay we use computers, for instance what am I going to 
show to students on the computer and what not to [AA: of course] I do not think that blackboard and 
chalk is that bad, I am telling you. 

97. DO: Blackboard and chalk are also technologies. 

98. GA: Well I say that for instance for science classes. 

99. SI: See, it is different for English. I prepare the picture and project it to the wall. But I draw the 
picture or by finding it somewhere [LK: you cut it, paste it] I cut it, paste it. This takes my time [. . .] But 
if the picture [slide] is on the board, okay I sit down, I ask kids and we talk about it. We finish with one 
[picture], and then we start talking about the other. You see what I mean? This all happens with one 
button. 



101. SI: Of course, I give children the structure first. I give [present] the topic, they all got it, we did the 
grammar, when we turn practicing we will articulate it [the topic?] with the pictures. Then I need it [the 
computer] with the projector. 

Hypothesis 

Based on the overall interview data, I form the hypothesis that teachers hold a transmission model of 
teaching and regard the computer within that philosophy. They view knowledge as something that can be 
assimilated and further regard themselves as the dispensers of that knowledge. Within such philosophy, 
computers are mainly perceived as efficiency devices, that is, helping teachers to save time in presenting the 
topic and in doing the practice problems. 

Computers in Action: Two Cultural Models at Play 

1. Computers in real life, computers in education. If we look at teachers’ statements regarding the use 
of computers in education, we see that they bring up the theme of ‘efficiency’ such as ‘saving time, making 
instruction easier, etc.’ as the central benefits of using computers in the classroom. This suggests that these 
teachers relate the ‘technology in the classroom’ directly to the ‘technology in real life.’ In other words, they 
transform the cultural model they hold about technology in general to the classroom setting and try to make 
sense of it within their current classroom practices. The following quote captures this idea. 

28. RO: Use of computers in education, enables us to reach our educational goals faster . I mean. . . well, 
umm technological developments that make our life easier will also make education easier, I mean, 
make easier to educate students, make instruction quicker, and will enable to teach more effectively . 

[...] 
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2. Transmission model of teaching. However, the data also suggest that seeing the computer as an 
efficiency tool is not only due to teachers’ overlapping cultural models; it is also influenced by their underlying 
conceptions of knowledge and learning. 

Let us consider stanza 41, which is quoted above. Here SI talks about using the computer to show 
students pictures, which she uses to have students to use their English in explaining them. She invokes the ideas 
of novelty effect and “leaning by implementation.” She refers to “learning by implementation” showing 
students the pictures on the computer, which underlines a view of learners as passive receivers. It is reasonable 
to argue that “learning by implementation” would mean quite different things to someone who conceptualizes 
learning as a process of active construction or participation within a community of learners. Since students like 
technology (a comment she made earlier), she assumes that they will be engaged in whatever practices they are 
involved with. In the conversation that starts with stanza 94 (owned by DG and ends with her stanza 10 1), we 
see that DG puts forward the idea that “the teacher should own the knowledge at the end'' and AA trying to 
build an argument that computers are not that threatening as long as “the preparation” is owned by the teacher. 

SI aligns herself with this uncontested idea (along with GA) by first clarifying the computer use as a tool that 
makes her classroom practices efficient. That is, computers are not that bad as they make the use of pictures 
easy. Then she clearly states that: “Of course, I give children the structure first. I give [present] the topic" and 
reveals a cultural model of teaching that underlines herself being the container and dispenser of knowledge. 

GA is actually the only teacher who articulates explicitly concerns about the compatibility of computer 
use with the student-centered educational philosophy (see stanzas 47 and 49). She accepts that using computers 
can help saving time; however she is not sure how they can be used in a student-centered way. Nonetheless, one 
cannot conclude that she is actually holding a student-centered perspective. In stanza 98, she finds a way to 
agree with DG. She says “Well, I say that for instance for science classes, " by finding the middle way with her 
statements before and with DG, who basically favors blackboard and chalk over computers, since blackboard 
and chalk help to maintain the teacher’s status as the center of knowledge. 

LK talks about using educational CD’s “after the topic is presented” (see stanza 61). Although she uses 
a passive structure there, and it may seem as we may not be sure who presents the topic — the teacher or the 
students, but the word that she uses for “presenting” in Turkish is more similar to “lecturing,” which is mostly 
attributed to teachers. She mentions the Internet; however she does not make any comments either here or in the 
rest of the interview for how she might utilize it. Her current teaching practices, as explained here, also do not 
seem to indicate the use of computers to engage students in any kind of exploration that could lead to 
construction of knowledge or participation. 

In stanza 75, we can see that AA clearly articulates a position as the center of knowledge and only sees 
computer as a supporting tool for his effectiveness: “[If I used computers] I would be more effective, give more 
knowledge [emphasis added], and save more time. " And we see GA also agrees with that statement, even 
though earlier she conveyed her concerns about the usefulness of computers in a student-centered classroom. AA 
further associates making students comprehend the topic better with doing more exercise problems in stanza 78. 
And for SI since technology is inherently interesting for students, she aligns her line of thought to this tread too. 
AA finally concludes that if he teaches with computers, “the situation would be better than the blackboard and 
chalk classroom.” However, the transition from blackboard to computers can be a better one not because it might 
help to change the traditional role of the teacher as the owner and dispenser of the knowledge; it is a better one 
since it is more efficient in sustaining this positioning of teachers’. 

This position is most clearly articulated by DG (stanza 94) with the sentence “The teacher should own 
the knowledge at the end.” As evident in his stanza 96, DG also views the computer as threatening the 
information center position of the teacher and favors the older technology (blackboard) as a way to maintain that 
status. He says “But we should be thinking about this, okay we use computers, for instance what am I going to 
show students on the computer and what not to. " These statements suggest that DG fails to realize other possible 
forms of couplings with computers and his teaching. 

Coverage and Convergence 

This hypothesis can also help us understand the other parts of the interview (coverage) and other parts 
continue to support the hypothesis as well (convergence). As stated earlier, these teachers reported that they 
were not incorporating computers in their classes much, although all of them consider themselves computer- 
literate. In addition, they articulated very favorable attitudes toward teaching with computers. Eventually, one 
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would expect that these teachers do teach with computers all the time. However, they do not. Later in the 
interview, they mention several micro-level and macro-level factors affecting their control over the use of 
computers in their classes. And they claim that it is these factors that are holding them back. However, I argue 
that the reason teachers do not incorporate computers in their teaching can still be explained by the hypothesis 
that teachers hold a view of knowledge as something that can be contained and transmitted, and a view of 
themselves as the center of knowledge. 

Fey (1989) describes several roles that teachers can enact in a technology-intensive learning 
environment. When technology is available, we might see shifting roles for the teacher from expositor and 
drillmaster to task setter, counselor, information resource, manager, explainer, and fellow student. It seems that 
the reason why these teachers have difficulty in seeing the role of the teacher as anything other than the 
information dispenser may be due to their cultural model of knowledge. For them knowledge is something 
contained in some kind of storage (either in them or in the computer) and this creates the tension of ‘either or,’ 
which in turn blocks them from benefiting from computers in ways other than presenting information. 

Moreover, their claim that infrastructure is the biggest issue that restrain their using computers in their 
teaching becomes weaker since in another part of the interview they displayed a lack of awareness about the 
technical means available in the school, which was evident in the following excerpt: 

249. DO: [As] we talked about it earlier, we talked for the question, everybody wants it [use the 

computer], finds it beneficial for student learning, and there are computers in your school as well 

250. AA and some other teachers: Yes that is our common view, we all want it. 

25 1 . LK and ZK: But, having access to the Internet. . . It is not open to every teacher, isn’t it? Is it open 

to every student or teacher? No. 

252. The computer teacher: It is [open to everyone] actually. And every computer in that room has an 

Internet connection. 

It is true that teachers’ indifference regarding the technical resources and means available at the school 
can be partly explained by the lack of technical support they receive. They might think that they would have to 
waste time to troubleshoot technical problems that would be either caused by low competence on computers or 
by insufficient technical equipment. But an alternative explanation appears to be that what teachers really think 
might be the irrelevance and/or inconvenience of computer implementation with their classroom practices or 
with their educational philosophy. Viewing computers merely as presentation devices, they may think that 
computers are not essential for their teaching style, since presenting the topic is something they attribute to 
themselves as teachers and they do not necessarily need the computer for this. 

Activity and Identity Building. Then why do these teachers choose to speak about computers this way? 
In order to answer this question, I need to address the kind of activity they and I were building, and kinds of 
situated identities they and I were enacting. As a person whose parents are either teachers or school 
administrators, I have spent a lot of time with similar group of teachers and engaged in similar conversations in 
non-formal settings in several times. In such situations, teachers do not necessarily enact their professional 
teacher identity. However, these teachers only knew me as a graduate student who was interested in computer 
use in their schools. I had a permission letter from the Ministry of National Education to do research in their 
school and we were involved in a school-based activity together. In order for me to recognize them as teachers, 
they needed to situate themselves in the teacher [capital D] Discourse (Gee, 1999), and use the related social 
language. During the interview they were enacting their professional identity as teachers by using the words of 
the teacher discourse, such as, educational goals, interaction with students, gap between students, low-level 
versus high level students, participation in class, visual, auditory methods, etc. In addition, they were most likely 
saying things that I would like to hear from them, such as 'learning with experience, interaction, etc. ’’ While 
interacting with me, they were employing the social language of teaching and constructing their identities as 
professional teachers. 

Teachers’ activity and identity building did not happen in a vacuum. I also participated in it, and the 
reader can recognize my identity construction as a researcher in this interview. I was acting on my researcher 
identity by stopping them whenever I felt they said things that I thought would not be relevant for my research 
questions and also providing them the vocabulary to talk about their experience, such as “motivating, 
entertaining, or beneficial.” Ultimately, my identity and activity building were interacting with those of the 
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teachers’, and mutually forming each other. Thus, teachers’ positive evaluation statements about teaching with 
computers need to find meaning within this joint activity and situated identity building. 

In summary, the main reason of not using computers in teaching is not related to merely infrastructure- 
related problems. Teachers’ positive evaluations are mostly due to the joint activity and identity building rather 
than being an indicator of what they really think about computers. Seen from this perspective, these positive 
statements about computers fail to imply an alternative explanation (That is, these teachers like computers, and 
they want to teach with computers, but infrastructure problems are restraining them). Therefore the discrepancy 
between these positive evaluations about teaching with computers and teachers’ actual classroom practices is 
mostly understandable on the basis of the hypothesis generated (coverage). In addition, other parts of the 
interview also continue to support the hypothesis (convergence) by refuting this alternative. More specifically, 
the hypothesis is also being supported by the fact that the teachers appear to be indifferent to the technological 
resources available at the school. 



Discussion 

In the educational realm, it is important to know about factors that could block or decelerate reform 
efforts. This becomes even more important in a context where a lot of funds have been allocated for reform. 
Burbules and Callister (2000) argue that reform will not simply follow from changing technologies in schools, 
since this is only a small part of educational reform. Along with technology, other educational practices and 
relations need to be changed for computers to play the desired role within reform efforts. The findings of this 
study corroborate this idea by identifying a less visible factor that functions as an obstacle to use computers in 
education and to realize the ideas of reform. The data suggest that teaching philosophies of the seven Turkish 
teachers operate as such a barrier. It is not, however, the purpose of this paper to criticize these teachers’ heavily 
behaviorist foundations in teaching, even though more recent educational reforms have advocated abandonment 
of behaviorist philosophies. Rather, following Sfard (1998) and Boaler (2002), what I want to emphasize is the 
dominance, if not the singleness, of this view that may be blocking reform efforts. 

The findings of this study are important as this study takes the discussion about reform obstacles to a 
new ground. Apart from issues related to technical access to computers and computer literacy, it is suggested 
that teachers need to become aware of and reflect on their teaching philosophies, a process which will help them 
to take different roles in technology-rich environments. With their current cultural model, teachers view 
computers only as efficiency devices, which help them to save time in presenting topics and in doing practice 
problems. If teachers, however, operated under alternative views on knowledge and learning, they would more 
likely design activities with computers differently. For instance, if teachers also viewed knowledge as the aspect 
of practice/discourse/activity, and learning as becoming a participant in a social context, they could move from 
being the center and dispenser of knowledge to acting more like a facilitator. Teachers would concentrate on 
designing meaningful activities in which students want to participate and computers can be used as tools to solve 
more authentic and challenging problems. 

One implication of the findings of this study is that both teacher education and teacher professional 
development programs need to address issues larger than computer literacy. While the administrative system 
constantly assures solving financial and technical level problems, teacher education programs and teacher 
professional development need to offer alternative models of knowledge and learning. This could help teachers 
to realize the potentials of the computer other than just being a presentation device, as the prevailing monopoly 
of the transmission model of teaching may hinder teachers to implement new technologies in a way envisioned 
by any reform ideas. 

Although the purpose of this study, by its nature, is not to make generalizations, I believe the findings 
are not unique to the Turkish context. We know that in almost every country, teachers do not embrace new 
technologies for instruction easily. ^ This study highlights the availability of less detectable yet significant 
factors that may help us understand why teachers in general are not adapting technology at desired levels. 

In addition to the findings, I would like to highlight the notion of ‘cultural model’ (Gee, 1999) as a 
powerful theoretical construct, as an alternative to the most common notion of ‘belief’ What leverage does one 
gain by calling teaching philosophies as cultural models rather than as belief si First, by identifying something 



^ Cuban (1986) documents several reasons for the U.S. situation. 
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as a cultural model, one is empowered with a complete theory about human sense making that reveals itself in 
the way we speak and orient ourselves to language. This theory informs us that we are usually unaware that we 
are operating under certain cultural models and their full implications. And unless they are challenged by 
someone or by new experience where our cultural models do not fit, we keep functioning under them. Second, a 
cultural model is distributed; bits and pieces of a cultural model are in people’s head, and other bits and pieces 
reside in the practices and settings of cultural groups. That is, this conceptualization highlights the role of a 
larger social context in the formation and sustaining the cultural models one has. Thus, while the notion of belief 
comes with a connotation of something personal and internal, the idea of cultural model draws attention to a 
larger social context, leaving one to question perhaps the whole educational culture. 
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Introduction 

Problem-based learning (PBL) has been shown to have positive effects on students enrolled in medical 
programs and, more recently, at the K-12 level. One potential benefit of using PBL in K-12 education is the 
improvement of students’ self-regulation skills. Self-regulation skills refer to “students’ self-generated thoughts, 
feelings, strategies, and behaviors, which are oriented toward the attainment of goals” within the learning process 
(Schunk & Zimmerman, 1998, p. viii). In this study, guided interviews were used to investigate the experiences of 
low self-regulated learners in a problem-based learning environment. Low self-regulated learners were identified 
through the Self-directed Learning Inventory (Metiri Group, 2002) completed at the beginning of the school year. 

Literature Review 

Problem-Based Learning 

Problem-based learning (PBL) is a student-centered instructional method driven by an ill-structured, 
realistic problem on which students collaborate in order to develop feasible solutions. PBL creates a learning 
environment where students are active in the learning process. Students propose investigative questions, explore and 
research relevant information, and suggest solutions (Lambros, 2004). In addition, the teacher assumes the role of 
facilitator or guide, assisting students through the learning process with prompts, guidance, and resources (Savin- 
Baden, 2003). 

Benefits for K-12 Students 

The majority of research on the PBL method has examined the benefits for medical students; however, 
emerging research lends support to the use of PBL in K-12 contexts. Evidence from current studies conducted in the 
K-12 environment has shown that using PBL instruction can increase student motivation (Koszalka, Graboski & 
Kim, 2001; Pedersen, 2003), problem solving skills (Ben-Chaim, Fey, Fitzgerald, Benedetto, & Miller, 1997), long- 
term retention (Albanese & Mitchell, 1993; Gallagher, 1997; Norman, 1991), self-efficacy (Cerezo, 2004), transfer 
of new skills (Pedersen & Liu, 2002), and collaboration skills (Achilles & Hoover, 1996). 

One potential benefit of problem-based learning is the improvement of students’ self-directed and self- 
regulation learning skills (Hmelo-Silver, 2004; Zimmerman & Lebeau, 2000). While these two terms often are used 
interchangeably to reflect student-chosen strategies for learning, emphasizing learner control over the learning 
process, one distinction between the two terms is the scope. Self-directed learning usually takes place with help and 
interaction from others (Knowles, 1975), while self-regulation refers to a more internalized process comprising 
“students’ self-generated thoughts, feelings, strategies, and behaviors, which are oriented towards the attainment of 
goals” (Schunk & Zimmerman, 1998, p. viii). For the purposes of this paper, we will use the term self-regulation 
due to its emphasis on the internal management of the PBL process. 

Self-Regulation in Problem-Based Learning 

Self-regulation is a characteristic commonly required for students to be successful in PBL contexts (Hmelo- 
Silver, 2004; Zimmerman & Lebeau, 2000), yet some authors argue that PBL also enhances students’ ability to self- 
monitor (Belfiore & Homyak, 1998; Lan, 1998; Zimmerman & Paulsen, 1995). According to Hmelo-Silver (2004), 
problem-based learning helps students gain knowledge while developing self-regulation strategies. While in a 
problem-based learning environment, students investigate a problem through research and questioning, establish a 
plan for solving the problem, and take the steps necessary to provide a workable solution. In order to successfully 
complete these tasks students must engage in these self-regulation skills throughout the entire process. Students use 
self-regulation skills while dissecting the problem as a group, extracting pertinent information, and creating a list of 
information they still need to find. Once the information has been divided among different group members, students 
use additional self-regulation skills in order to investigate different resources, bring the information back to the 
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group, and evaluate the avenues that still need to be explored or that need additional information (Torp & Sage, 

2002) . All of these activities within a problem-based learning environment call for self-regulation skills. “The SDL 
[self-directed learning] emphasis is a distinguishing feature of PEL. In PEL, students become responsible for their 
own learning, which necessitates reflective, critical thinking about what is being learned” (Hmelo-Silver, 2004, p. 
239). Problem-based learning environments are learner-centered, asking students to apply knowledge they learn in a 
self-regulated manner. This leads to one of the overall purposes of engaging students in a PEL unit; that is, to build 
students’ ability to self-direct their own learning and become life-long learners (Hmelo-Silver, 2004). 

Self-regulation skills may influence students’ abilities to solve problems (Swanson, 1990). Students with 
self-regulation skills possess the ability to “select, control, and monitor strategies needed to achieve desired learning 
goals” (Ertmer & Newby, 1996, p. 1). When high self-regulated learners approach a topic they do not understand, 
they are better able to apply strategies to learn the information than a learner with lower self-regulated skills 
(Swanson, 1990). However, some research (Eelfiore & Homyak, 1998) has shown that students of low-achieving 
status, who also tend to be low self-regulated, have the ability to successfully solve problems and participate in 
complex learning activities, such as PEL units. However, they usually require more guidance for setting their own 
goals and monitoring their own growth, which can be accomplished through such scaffolding tools as checklists 
(Eelfiore & Homyak, 1998). Although some studies have shown a possible correlation between low-achievement 
and low-self-regulation (Salili & Lai, 2000), others have found no significant differences between high and low 
achieving students and their use of self-regulated strategies (Rao, Moely, & Sachs, 2000). Further research is 
required to investigate this possibility, especially at the K-12 level (Chuang, & Pape, 2003). 

One difficulty with PEL is the challenges it provides for students with low levels of metacognition, an 
important factor in self-regulation (Hmelo & Lin, 2000). “Metacognition is essential to function well in complex 
environments and therefore [students] must be supported in developing a sense of responsibility for their 
management of problem-solving tasks” (Greening, 1998, p. 7). A study by Swanson (1990) indicated that high 
metacognitive ability correlated with a student’s ability to solve problems, regardless of aptitude. Therefore, it is 
important to equip all students, regardless of ability, with metacognitive skills (Swanson, 1990). In another study, 
low-achieving students performed at, or above, the level of average achieving students within anchored instmction, 
one type of problem-solving environment. The authors attributed these results to five main components of anchored 
instmction: the motivation stemming from working on real problems, students incorporating prior knowledge and 
being metacognitive, the collaborative environment, the guidance provided by expert teachers, and the mini-lessons 
incorporated to supply students with the necessary content knowledge (Eottage, Heinrichs, Chan, Mehta, & Watson, 

2003) . Although research has examined low-achieving K-12 students in different educational environments 
(Eottage, et al., 2003; Eelfiore & Homyak, 1998; Swanson, 1990; Eom & Reiser, 2000), research is limited on how 
low self-regulated K-12 students manage learning, especially while in PEL environments. This study investigated 
the perceptions of low self-regulating students regarding how they managed their learning within a problem-based 
learning environment. 

Purpose 

Problem-based learning (PEL) usually requires greater student responsibility and activity than traditional 
learning (Hmelo-Silver, 2004). Eecause low self-regulated students may lack the ability to guide their own learning 
(Swanson, 1990), they may have difficulties within a PEL environment. It is important to understand the strategies 
that low self-regulated students use in a PEL environment in order to design future environments that enable success 
for these students. In order to study the strategies and experiences of students with low levels of self-regulation in a 
problem-based learning environment, we posed the following research questions: 

1 . What are the experiences of low self-regulated learners in problem-based learning environments? 

o How do low self-regulated learners manage the learning process in a problem-based learning 
environment? 

o What do low self-regulated learners perceive they learned in problem-based learning 
environments? 

2. What are the perceptions of low self-regulated learners of the benefits and challenges of problem-based 

learning? 

Method 

Participants and Context 

This exploratory study was designed to investigate the experiences and perceptions of low self-regulated 
learners in a problem-based learning environment. The five males and five female students selected for this study 
were all six-graders from a rural district in the Midwest and participated in the same problem-based learning units 
with one of two collaborating language arts teachers. Students for this study were selected based on their self- 
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directed learning scores taken at the beginning of the year (details regarding selection procedures are presented 
later). 

The two Language Arts teachers collaborated and co-planned three PBL units during the year, one 
implemented each trimester. In general, the Language Arts standards being met through these units addressed 
persuasive communication skills, writing skills, research/note taking skills, and oral presentation skills. The first 
problem-based learning unit placed an emphasis on improving the local community. Students were grouped and 
charged with selecting a problem in their community, creating a change/solution, and presenting their proposal to a 
community official. The second problem-based learning unit was based on converting the old library into a museum. 
Students developed an exhibit idea and competed to have their exhibits displayed in the museum. Students were 
required to individually create their own museum exhibits and to produce a persuasive argument detailing why their 
design should be chosen. Students were then grouped together in their math classes and each group selected one of 
the ideas to develop further. Once the idea was selected, students created a blueprint of their exhibits using scales 
and proportions. The final problem-based learning unit addressed the deforestation of rainforests. The students 
selected one of eight reasons why they wanted to save the rainforests, ranging from the medicines it provides to the 
possible extinction of certain animal species. They separated into groups based on the reason they selected. The 
groups researched their reasons for saving the rainforest, created a presentation, and defended their stance to their 
classmates. 

Procedures 

In fall 2004, all sixth grade students (n=152) completed the Self-Directed Learning (SDL) inventory (Metiri 
Group, 2002) prior to participating in any of the PBL units. The instrument measured the following self-rated skills: 
goal setting/strategic planning, self instruction/imagery (forming mental pictures), self monitoring, help seeking 
(self-initiated attempts to receive assistance), self evaluation, and adaptivity (adjusting when confronted with errors). 

Students with a low SDL score, defined as at least one standard deviation below the mean, were identified 
as possible students to interview. The research team consulted with the teachers to select students who would be 
willing to speak out and who could afford to miss a portion of class time. Six students were interviewed in the 
middle of the school year after completing two problem-based learning units while the other four students were 
interviewed at the end of the year after the third problem-based learning unit had been completed. 

The selected students were interviewed by one of the first two authors using a semi-structured protocol. 

The interviews lasted for ten to twenty minutes. Questions covered their attitudes towards PBL, projects they 
completed, and descriptions of the learning process. For instance, one question asked, “Do you like these PBL 
projects more or less than normal class activities?” Student interviews were transcribed and coded based on key 
words related to our research questions, such as group interactions and attitudes towards PBL. 

Results and Discussion 

Experiences in PBL 

When low self-regulated students discussed their experiences in a problem-based learning environment, 
common trends were found throughout the interviews. Students discussed how they managed the learning process, 
their perceptions of the benefits and challenges to working in a PBL environment, including working in groups, as 
well as their perceptions of what they had learned. 

Managing the process. Based on the first research question, common trends were found among low self- 
regulated learners’ perceptions of their experiences in problem-based learning environments, specifically concerning 
the management of their own learning. Students with low self-regulated skills turned to a variety of support 
mechanisms during the problem-based learning units, although asking the teacher for help was most frequently 
mentioned: eight of the ten students mentioned using this strategy. One student recalled asking the teacher for 
assistance he/she could not move forward; “I had to ask my teacher. She fixed it.” Five students described relying 
on their own problem-solving abilities. One student described how he overcame being stuck on a problem by “just 
using [his] brain.” When he was stuck or confused, he described reasoning the situation out in his head by thinking 
of alternative solutions to the problem. 

One potential concern for students within PBL relates to those who lack organizational and monitoring 
skills, or do not complete assignments on time (Belfiore & Homyak, 1998). Out of the ten interviewees, five 
students either had not turned an assignment in or turned it in late. Students who did not complete assignments on 
time attributed such behaviors to their frequent absences or merely being uninterested in the completion of the 
project. One student attributed the lateness to his “forgetting my laptop at home,” while another student attributed 
the lateness to, “I just didn’t [put] the time into it. I did it at the last second. [I was] just trying to get it completely 
finished.” It is plausible this stemmed from a lack of self-regulation, because self-regulation skills include 
organizational and monitoring components. 
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This result is supported by the literature; researchers have noted that students with low self-regulated skills 
often have a difficult time finishing assignments due to lack of organizational and monitoring skills (Zimmerman & 
Kitsantas, 1997). Additional guidance and scaffolds may enable these students to accomplish their goals in a 
problem-based learning environment (Schunk & Ertmer, 1999), which could be achieved through the use of more 
checkpoints throughout the process. 

Another part of managing the process within problem-based learning requires metacognitive skills. 
Metacognition enables students to think about their own learning, the strategies they are applying, and what they can 
do to adapt the process when it is not working. Typically, low self-regulated students lack the metacognitive abilities 
to direct the learning process and take charge of their own learning (Swanson, 1990). 

However, students did occasionally appear to use metacognitive skills. Seven students mentioned they 
realized the need to search for more information to prepare an adequate presentation. Another student noted she 
needed to think of the situation in an alternative way to solve the problem. The student stated the need for thinking 
harder by saying to herself, “think of something else, think of something else.” This evidentiary support suggests 
that some students were capable of using metacognitive skills, which is a promising sign for their success in future 
PEL experiences. 

Perceptions of learning concepts. During the interviews we also asked students to describe what they 
learned from their problem-based learning experiences. Throughout the problem-based learning units, students 
worked on both mathematics concepts related to proportions and scales, as well as language arts standards related to 
communication, research, oral presentations, and persuasive writing. Most low self-regulated students lacked the 
ability to articulate their understandings of the specific concepts taught and learned in language arts, although in 
mathematics they were able to refer to specific learning goals using the appropriate mathematical terms. We suspect 
this was due, in part, to the difference in subject structure; where mathematics (a “hard” science) instruction is 
typically with precision, language arts (a “soft” science) is more viewed as more uncertain and open (Ambrose, 
1998). It may be easier for students to recognize the specific goals of mathematics, as opposed to the more open 
goals of language arts. 

Through informal discussions with the teachers, another possible reason for the large variance was that the 
math teachers were new to PEL so they approached the unit using more direct instruction. The language arts 
teachers had previously used PEL, and consequently were more comfortable using more open-ended instruction 
where goals were not specifically reiterated on a daily basis. 

In the mathematics unit students drew blueprints and learned how to calculate proportions and configure 
scale drawings. Half (n=5) of the low self-directed students were able to specifically articulate the processes learned 
in math class, explaining how to calculate proportions. “Like, what it is - 2 squares equal — 1 inch equals 2 squares. 
And if you had 20, decrease to 10.” Other students had less specific descriptions of the content learned such as 
“making drawings” (n=4), “measuring” (n=3), and “blueprints” (n=l), but still used appropriate mathematical terms. 

In language arts, students had more difficulties articulating specific concepts they had learned from the 
problem-based learning units. These units were designed to help students learn how to write a persuasive argument, 
give an oral presentation, and write a research paper. Three students actually mentioned they “haven’t learned 
anything” related to language arts. Some students (n=4) mentioned they believed they learned how to research more 
proficiently, while other students noted that they learned how to make decisions (n=l), how to revise his/her writing 
(n=l), how to gather information (n=l), how to use bigger words (n=l), how to type faster (n=2), the importance of 
providing details and supporting writing with arguments (n=l), different grammatical elements (n=2), how to create 
a PowerPoint presentation (n=2), and what is PEL (n=2). 

While more students were able to articulate their understandings of the mathematics content learned, few 
students were able to state the goals of the language arts units. Ey not having a well-defined learning objective, 
students may have been more confused and uncertain about the tasks they needed to complete in the problem-based 
learning environment (Abrandt Dahlgren & Dahlgren, 2002). 

Perceptions of Eenefits and Challenges 

The second research question investigated the attitudes of low self-regulated learners toward problem- 
based learning. All students interviewed (n=10) indicated that they preferred PEL over traditional instruction. 
Reasons included: “it’s easier,” “we have a choice to do whatever we want,” “I just like working with partners,” and 
“I’m getting good at solving problems.” 

While students listed attributes of PEL they enjoyed, there were aspects that they perceived made learning 
more challenging and difficult than traditional instruction. Low self-regulated students noted frustration due to the 
large amount of time required (n=5), the difficulty of finding information (n=6), and the difficulty level of the 
project (n=5). These could all possibly be attributed to the students’ lack of self-regulation. The frustrations relating 
to the time commitment may be because it is difficult to manage a complex project if students do not have strong 
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time management skills. The last frustration pertaining to the difficult nature of the project may be due, in part, to 
the role self-regulation plays throughout the whole problem-based learning process. Therefore, students with lower 
self-regulation skills may not be able to properly manage the project in its entirety, making it more difficult than 
traditional projects. We believe this frustration may be attributed to the lack of self-regulated strategies required in a 
problem-based learning environment, although more research is necessary to investigate this conjecture. 

Group work. Another key component of problem-based learning is working in groups. In this study 
students were placed into groups of three to four in at least two of the units. Almost all students (n=8) preferred 
group work over individual work. When asked about their group experiences, student responses included “that 
makes it a lot easier cause you can split up the work” and “I like spending time with friends in my groups.” 

There were many positive aspects of group work that students mentioned throughout their interviews. Half 
of the students (n=5) referenced the help they received from group members as being important to their success and 
enjoyment. One student remarked, “I think [working in a group] is a lot better than working by yourself because you 
have other people that understand it more.” 

The reason that most of the low self-regulated students in this study preferred groups could be due to the 
age of the students, who desire more opportunities for social interaction (Lambros, 2004), or because the 
collaboration provides them with the opportunity to receive assistance from others. Regardless of the reason, low 
self-regulated students did appreciate the collaborative effort of working towards a goal. O’Connor and Jenkins 
(1993) proposed that teachers should replace individual assignments with more small heterogeneous group work to 
provide more support for struggling students. 

Conclusion 

Self-regulation is important to students’ success in problem-based learning environments (Hmelo- Silver, 
2004) and those without the necessary self-regulation skills may not flourish in these more open-ended 
environments. Overall, low self-regulated learners have difficulties participating in complex learning environments, 
such as those involving problem-based learning, due to a lack of ability to plan, monitor, and evaluate their learning 
(Ertmer, 1995). The low self-regulated learners in this study expressed specific difficulties thinking of alternate 
learning strategies, monitoring the project to finish on time, engaging in more ill-defined problems with confidence, 
planning their learning processes, and dealing with the challenging nature of problem-based learning. Low self- 
regulated learners need support, guidance, and assistance to manage projects in a given time frame in problem-based 
learning classrooms (Biemiller et al., 1998). Therefore, we suggest the use of tools such as checklists, worksheets, 
webquests and recommended websites, notecards, and guiding questions to assist low self-regulated learners 
(Biemiller et al., 1998). With the proper fading of such guides and instruction on metacognitive skills, low self- 
regulated learners may be able to obtain more self-directedness in such open-ended learning environments (Belfiore 
& Homyak, 1998). At first, it may be necessary for students to receive “closer teacher attention” when using self- 
regulated strategies (Mason, Harris, & Graham, 2002). One interviewee said she now performs many of the same 
strategies in other classes because it helps her with researching and managing her learning. If possible, we 
recommend collaborating with team teachers to help implement these strategies in every learning situation (Mason 
et at.). By equipping low self-regulated students with these skills, they will be better able to manage the learning 
process on their own. 
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Abstract 

Positive emotion plays a critical role in learning process. Learning interest among positive 
emotions, especially, initiates learners’ motivation and further improves learners’ achievement. 
This paper explored two research questions based on the distinction between individual interest 
and situational interest: (1) what factors affect learning interest in pedagogical agents embedded 
learning, and (2) How to design pedagogical agents to improve learners’ interest and attitude. In 
addition, this paper further investigated the effects of pedagogical agents developed using factors 
identified above on learning interest, achievement, and attitude. 



Learning Interest 

Interest increases learning. Promoting interest in the classroom increases learner’s learning motivation and 
improves the use of learning strategies. (Keller, 1983; Pressley, El-Dinary, Marks, Brown, & Stein, 1992; Sweet, 
Guthrie, & Ng, 1997). Interest also plays a significant role in the learning process, determining what to learn, and 
how to learn the chosen information (Alexander & Jetton, 1996; Gamer, 1992). Therefore, the issue of how to 
increase learner’s interest is directly related to initiating learner’s motivation, and further improving learner’s 
achievement. 



The Framework of Research on Interest 

Renninger et al (1992), in their book entitled “The role of Interest in Learning and Developmenf’, 
provided an integrated theoretical framework based on the distinction between individual interest and situational 
interest. This framework has been used as a theoretical ground for studies on learning interest. 

The distinction between individual interest and situational interest has been drawn as a framework in many 
empirical studies on interest (Gamer et al., 1989; Gamer et al., 1992; Harp & Mayer, 1997, 1998; Hidi, 1990, 2001; 
Hidi & Anderson, 1992; Hidi & Baird, 1986; Krapp, 1999, 2002; Krapp et al., 1992; Schiefele, 1991, 1992, 1996, 
1998, 1999; Schiefele & Krapp, 1996; Schraw, 1998; Schraw et al., 2001; Schraw & Lehman, 2001; Wade et al., 
1993). From the view of individual interest, interest is implied as a characteristic of person. It is specific to 
individuals, developed slowly, tends to be long lasting, and is triggered by an individual’s predisposition (Renninger 
et al., 1992; Schiefele, 1998; Schraw & Lehman, 2001; Silvia, 2001). For example, learners who are interested in a 
topic or an activity pay more attention and acquire more knowledge than participants without such interest. 
Individual interest can be assessed through a learner analysis process by asking several background questions or 
administering a simple questionnaire. However, it is not easy to take into account individual interest in designing 
learning material because individual interest refers to a student’s relatively enduring preference for different topics, 
tasks, or contexts (Krapp, 1999; Tobias, 1994). 

Krapp (1992) insisted that situational interest be a prior foundation of individual interest. According to 
him, when the content of the learning material is not a subject area in which the learner has established individual 
interest, the interesting factors in the subject learning situation is necessary to awake the interest for a short or longer 
period of time. This interest is defined as situational interest and the central psychological process “Internalization” 
supports the transformation process of situational interest into long-lasting individual interest as described in Figure 
1 . Situational interest is generated as a result of interestingness of situation. It is caused primarily by certain 
conditions and concrete objects in the environment, triggered by environmental factors, elicited by certain aspects of 
a situation, and it is assumed to contribute to the interestingness of the situation (Harp & Mayer, 1997; Hidi & 
Anderson, 1992; Krapp, 1999, 2002; Renninger et al., 1992; Schiefele, 1992, 1996; Schraw & Lehman, 2001; Silvia, 
2001). While individual interest is a relatively stable evaluative orientation towards certain domains, situational 
interest is formed if an emotional state aroused by specific features of an activity or a task. The following Figure 1 
illustrates the relationship between individual interest and situational interest. This study grounded its framework of 
inquiry in this relationship. 
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As illustrated in Figure 1, the relationship between individual interest and situational interest is hardly 
clear. Both individual interest and situational interest are combined together in certain learning circumstances to 
represent the interest as a psychological state within a person. 




Figure 1. Three approaches to interest research (Krapp et al., 1992). 

Individual Interest 

Individual interest is topic specific and persists over time (Midi, 1990; Krapp et al., 1992). Individual 
interest is defined as people’s relatively enduring preferences for different topics, tasks, or contexts (Krapp, 1999; 
Tobias, 1994). Schiefele (1991, 1999) further distinguished between two subcomponents of individual interest; a 
feeling-related component and a value-related component as illustrated in Figure 2. 

Feeling-related valences refer to the feelings that are associated with an object or an activity, for instance, 
involvement, stimulation, and flow. It occurs when an individual experiences positive affect and emotions in 
conjunction with a particular topic or activity. It is presumed that these positive feelings provide a strong 
motivational incentive to engage in the activity (Schraw & Lehman, 2001). Value related valences refer to the 
attribution of personal significance of importance to an object and activity. It is presumed that value related interest 
increases engagement because an activity or body of knowledge is judged to be salient to one’s goal (Sansone, Weir, 
Harpster, & Morgan, 1992; Schraw & Lehman, 2001). Since those two aspects are highly correlated each other, it is 
hard to determine what aspect of components affect individual interest (Schiefele, 1991, 1999). However, it is 
obvious that some learners prefer to learn specific topics or participate in specific activities because they like it 
primarily based on feeling, and others also do because they think the activity is important for some future purpose. 

Situational Interest 

Situational interest is elicited by aspects of a situation, such as novelty or intensity, and by the presence of 
a variety of human interest factors contributing to the attractiveness of different types of contents (Krapp, 1999; 
Tobias, 1994). Most of the research on situational interest has focused on the characteristics of academic tasks that 
create interest (Midi & Baird, 1986). Features that have been found to arouse situational interest and promote text 
comprehension and recall are: personal relevance, novelty, activity level, and comprehensibility. Situational interest 
is often considered to precede and facilitate the development of individual interest (Krapp et al., 1992). 

Most interest research has focused on various aspects of situational interest. As Schraw and Lehman 
(2001) indicated, one of the reasons is that situational interest is more amendable to change compared to individual 
interest. They further categorized three general aspects of situational interest as text-based, task-based, and 
knowledge based. Text-based interest refers to the feature of the text information that affects interest. Task-based 
interest involves task manipulations or encoding instructions that increase interest. Finally, knowledge-based interest 
refers to the aspects of learner’s knowledge-base that promote interest. Seductive detail, which refers to the highly 
interesting but unimportant text segments, is considered one of the factors in text-based aspects of situational interest. 

Regarding situational interest, the concept of cognitive interest and emotional interest was proposed by 
Kintsch (1980). Cognitive interest adjuncts such as explanative summaries, influence learner’s cognition by 
promoting the reader’s structural understanding of the explanation. On the other hand, emotional interest is 
explained by that the addition of interesting but peripherally relevant material to a textbook lesson that energizes the 
learner so that they pay more attention and learn more overall (Harp & Mayer, 1997). According to the definitions of 
cognitive interest and emotional interest, these are interest state derived by specific feature of a learning material. 
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Therefore, both of cognitive interest and emotional interest need to be counted as two aspects of situational interest 
as shown in Figure 2. 





Figure 2. Individual interest and Situational interest. 

Pedagogical Agent and Learning Interest 

Pedagogical agents are animated life-like characters designed to facilitate learning in computer-mediated 
learning environments (Johnson et ah, 2000). Regarding the effects of pedagogical agents, researchers using lifelike 
pedagogical agents have indicated positive effects on learner’s attitude toward learning and performance (Baylor, 
2002a, 2002b; Baylor & Ryu, 2003; Moreno et al., 2001). Also researchers have found positive effects of 
pedagogical agents on learner’s interest and motivation. Two explanations were suggested in regard to the positive 
effects of pedagogical agents on learner’s interest (Baylor & Ryu, 2003). The first reason was agent persona. 
Learners tend to like a pedagogical agent because of the agent’s character or personality based on agent’s persona 
composed of image, animation, voice, and emotional expressions. The second reason was the information provided 
by agent. Learners reported that they liked an agent because they found the quality of the message delivered by the 
agent was informative and helpful. The research using agents with evasive emotion and no motivational support also 
found that learner’s interest was aroused from the agent’s persona and information delivered by agent (Warren et al., 
2004). Therefore, the variables affecting learner’s interest could be categorized into two dimensions, one is an 
agent’s persona and the other is information delivered by the agent. In this paper, the variables affecting learner’s 
interest are described in detail in terms of the persona of pedagogical agents and the message delivered by the 
pedagogical agent. 

Pedagogical agent’s persona 

The key characteristics that constitute a pedagogical agent’s persona include its propensity to be engaging, 
person-like, credible, and instructor-like (Baylor & Ryu, 2003). Engaging personas of agents facilitate the learner- 
agent relationship and motivates the learner to be involved in the learning task. Person-like personas of agents form 
a viable relationship with the learner. Credible personas of agents make learners confident in the agent and help 
them recognize the agent as trustworthy, competent, and consistent in behavior. Finally, instructor-like personas of 
agents serve as a pedagogical mentor to effectively represent the content and pedagogy (Baylor, 2000). Media 
features such as voice, emotional expression, gesture, image, and animation are integrated to create such personas. 

The effect of a pedagogical agent on interest is explained also by social presence theory. According to 
Short, Williams, and Christie (1976), social presence is a subjective quality of the communication medium and can 
be a function of both verbal cues (e.g. tone of voice) and nonverbal cues (e.g. facial expression, direction of gaze, 
posture, and dress). Hence, a pedagogical agent is considered to have higher social presence than other computer 
mediated media, because pedagogical agents provide nonverbal cues as well as verbal cues, which can lead to 
promoted learner interest. 
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Therefore media features are essential in terms of constructing agent personas and indicating social 
presence of an agent to promote learning interest. Voice has been suggested as a key aspect for enhancing agent 
persona. In fact, the voice feature has been found to be a critical element for agent-based learning environment. Prior 
research has indicated that voice has a superiority effect to visual appearance for communication in computer-based 
media (Mayer & Moreno, 1998; Moreno & Mayer, 1999), and with pedagogical agents in particular. While there are 
consistent results that voice (in conjunction with text) is a key aspect for enhancing agent persona, the role of agent 
image and animation is not proven (Baylor & Ryu, 2003). However, given the importance of emotional expression 
by an agent, animation would be necessary for an agent to demonstrate facial expressions. Animation with emotional 
expression increases the agent’s persona of credibility and also improves social presence of the agent by increasing 
non-verbal behavior. Baylor (2003) found that animation provides the most positive impact for an agent to be 
perceived as engaging, because it contributes to the agent expression of a personality through non-verbal behavior, 
leading to be more likable, and thus more enjoyable to learn with and also more emotionally expressive. 

Messages delivered by Pedagogical agent 

Agents also facilitate interactive learning with learners by delivering messages to convey information. 

The type of information delivered by a pedagogical agent through messages is also a variable that affects learning 
interest. Information delivered by agent could be directly related to the learning content by signaling the structure of 
the learning content so that learners can promote the understanding of the content. Also, information could be 
interesting but irrelevant to understanding the learning content. According to Kintch (1980), information has an 
effect on the learner’s interest in two different ways depending on the features of information and he suggested the 
concept of cognitive interest and emotional interest. The cognitive interest influences learner’s cognition by 
promoting the reader’s structural understanding of the content. On the other hand, emotional interest is explained by 
the addition of interesting and irrelevant material to understand the objectives of instructional material. However, it 
energizes a learner’s arousal so that they pay more attention and learn more overall (Harp & Mayer, 1997, 1998). 
Researchers have used the term seductive detail to refer to interesting but irrelevant details that are added to a 
passage to make it more interesting in the reading education field. In multimedia learning, seductive detail usually is 
called seductive augmentation because the term “seductive augmentation” refers to not only text but also graphics, 
narratives, voice, animation, and text accompanied in a multimedia learning environment with the purpose of 
increasing learner’s situational interest (Thalheimer, 2004). 

As being illustrated in the previous sections, two opposing suggestions have been indicated regarding the 
usefulness of seductive detail. The first suggestion asserted that it energizes readers so that they pay more attention 
to learning and learn more overall, because it influences the learner’s affect by promoting his/her enjoyment of the 
topic. Hence, it causes the learner to pay more attention to and encode more of the information from the material 
(Izard & Ackerman, 2000; Kintsch, 1980). The second suggestion contends the opposite position. It emphasizes that 
seductive detail disrupts the learner’s construction of the material comprehension so that adding it to the material 
will result in decreases on tests of retention and on solutions to transfer problems (Wade & Adams, 1990). 

In summary, both agent itself including agent persona and social presence, and the information delivered 
by the agent plays a critical role in promoting learning interest from the aspect of situational interest. In particular, 
media features are important factors for constructing an effective agent persona. Information type is also a critical 
factor because it determines the quality of information delivered by agent. Figure 3 illustrates the detailed 
relationship describing the features of a pedagogical agent affecting learner’s situational interest. 
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Figure 3. The features of a pedagogical agent affecting learner’s situational interest 

Method 



Participants 

A total of 123 college undergraduate students enrolling in a computer literacy classes in a large public 
university located in the southeastern area participated in this study. Students were voluntarily participating in the 
study as an optional activity of class instruction. 

Independent Variable 

Four levels of the source of seductive messages were implemented in this study. The first source of 
seductive messages was the companion role of agent. The second source of seductive messages was the instructor 
role of agent. Both roles of agent were employed using two pedagogical agents. The third source of seductive 
messages was a text message without an agent. The last condition employed no seductive message and was 
considered as a control condition. The contents and length of the seductive messages were identical across the four 
different conditions 

Dependent Variable 

Dependent variables included post interest, achievement, and attitude toward intellectual property module 
supported by a pedagogical agent. To determine the validity of achievement test items, pilot test was conducted. The 
purpose of pilot test was to determine the level of prior knowledge of participants on “intellectual property” and to 
find a size of variance among participants. 

Interest measurement: Individual interest was measured from two aspects: feeling-related interest and 
value-related interest. The participants were asked to use the following adjectives in estimating their expected 
feelings (“While studying the instructional module on “Intellectual Property”/ 1 expect to feel. . .”) “bored”, 
“stimulated”, “interested”, “indifferenf ’, “involved” and “engaged.” When estimating value-related interest, 
participants were asked to use the terms “meaningful”, “unimportanf ’, “useful”, and “worthless” to describe the 
value of the topic “Intellectual Property” to them personally. All items used 5 rating scales (ranging from 
“completely true” to “not at all true”). Situational interest was measured from three aspects: arousal level, 
involvement, and attention. In order to measure the arousal level, AD- ACL (Activation-Deactivation Adjective 
Check List) was used (Thayer, 1985, 1986) with modified 5-likert scale. 

Involvement level was measured with intensity dimension (Reynolds, 1992) by two 5-likert scales such as 
“I was completely caught up in what I was studying” and “When learning from the module, I was concentrated”. 
Lastly, “attention” sub scale from IMMS (Instructional Material Motivation Survey) will be used to measure 
attention level of participants (Keller, 1993). 

Achievement measurement: Achievement test was designed to assess the learner’s ability to solve the 
given problems using what they have learnt from the instructional module. To determine the validity of test items, 
pilot test was conducted Students participating in pilot study were given eleven test items on the first day of 
computer literacy class and asked to answer the questions without any information on “intellectual property.” In 
main study, achievement test was given to measure learner’s achievement at the end of the module and graded based 
on pre determined answer sheet. Total score ranged from 0 through 1 1 . 
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Attitude measurement: The student’s attitude toward to the pedagogical agent supported multimedia 
learning was measured using IMMS developed by Keller (1993), to measure situational components of the ARCS 
model. Since attention part was used to measure the part of learning interest, the remaining components such as 
relevance, confidence, and satisfaction, were measured. The IMMS included 36 items intended to be a situational 
measure of learners’ motivational reactions to instructional material. The response scale ranged from one to five with 
nine relevance component items, nine confidence component items, and six satisfaction component items. The 
reliability of IMMS based on Cronbach’s alpha for each subscale was Relevance: .81, Confidence: .90, and 
Satisfaction: .92. 



Results 



The effects of agent role on learning interest 

A MANOVAanalysis revealed that there was a statistically significant difference of learning interest score 

71 ^ 

among the four seductive message conditions, Wilks’ Lambda=.709, F(15,318)=2.81, p<.001, ' =.11. Follow-up 

71 ^ 

ANOVA indicated that significant differences occurred in arousal, F(3, 1 1 9)=4.98, p< .0 1 , ' =. 1 1 , and attention, 

77 ^ 

F(3,l 19)=7.01, p< .001, ' =.15. Tukey HSD follow-up procedure indicated that arousal score for the companion 

agent (M = 2.19, SD = .57) was significantly higher than both the text only condition (M = 1.76, SD = .60) and the 
no message condition (M = 1.77, SD = .60). Also, arousal score for the instructor agent (M = 2.19, SD = .68) was 
significantly higher than both the text only condition (M = 1.76, SD = .60) and the no message condition (M = 1.77, 
SD = .60). For attention, the result indicated that the attention score for the companion agent (M = 3.30, SD = .65) 
was significantly higher than both the text only condition (M = 2.90, SD = .47) and the no message condition (M = 
2.76, SD = .66). Also, attention score for the instructor agent (M = 3.30, SD = .59) was significantly higher than 
both the text only condition (M = 2.90, SD = .47) and the no message condition (M = 2.76, SD = .66). 

The effects of agent role on achievement 

Recall test: An ANOVA on recall test scores revealed that there was no significant effect for the source of 
seductive messages, F(3, 119)= 1.37, p = .27. In order to examine the difference of total recalled number of keywords 
between both agent conditions and text only condition, a contrast analysis was performed. The results indicated that 
students in the agent conditions recalled significantly higher number of total keywords than did students in the text 
only condition, t(90) = 1.99, p < .05. Comprehension test: An ANOVA on comprehension test scores revealed that 
there was no significant effect for the source of seductive messages, F(3,119)=.52, p = .67. 

The effects of agent role on attitude 

A MANOVA revealed that there was a statistically significant difference of students’ attitude score among 

four seductive message conditions, Wilks’ Lambda=.782, F(9,285)=3.37, p<.01, ^ =.08. Follow-up ANOVA 

77 ^ 

indicated that significant differences occurred in relevance, F(3,l 19)=5.98, p< .01, ' =.13, and confidence, 

77 ^ 

F(3,l 19)=4.26, p< .01, ' =.10. Tukey HSD follow-up procedure indicated that relevance scores for the companion 

agent condition (M = 3.80, SD = .75), the instructor agent condition (M = 3.67, SD = .67), and the text only 
condition (M = 3.68, SD = .46) were significantly higher than the no message condition (M = 3.215, SD = .56) 
respectively. For confidence, the result indicated that confidence score for the companion agent condition (M = 3.30, 
SD = .57) was significantly higher than the no message condition (M = 2.82, SD = .67). 

Discussion 

Effects of seductive messages on learning interest 

The students who were given seductive messages delivered by a pedagogical agent earned significantly 
higher scores in arousal and attention in terms of learning interest than the students who were given either the text 
type of seductive message or no seductive message. Students in the no seductive message condition were not given 
any type of seductive messages, thus their level of learning interest was lower than the other conditions. An 
interesting finding is that there was significant difference of the arousal score and attention score between agent 
delivered seductive message and text based seductive message. The result of contrast analysis confirms that 
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seductive messages are more effective to promote students’ learning interest when they were delivered by a 
pedagogical agent rather than in text. 

There are at least two possible explanations for why the text type of seductive message was not as 
effective as the one delivered by an agent. 

First, the presence of an agent could affect students’ learning interest. Previous research on the effects of 
pedagogical agent showed that users’ interactions with the computer are much smoother when likable animated 
pedagogical agents are present. They also help students develop an emotional connection with the agent and 
facilitate their enjoyment of the learning situation (Dehn & Mulken, 2000). Therefore, although identical seductive 
messages were presented to both agent conditions and text only condition, students with agents could have been 
more interested in the learning of instructional material. 

The second reason is that the source of the messages was different in this study. In other words, seductive 
messages delivered by a pedagogical agent were audio messages via agent voice that is very different from text type 
of message. Therefore it is also possible that students perceived a sound type of seductive message more positively 
then the text type of message. The second reason was not confirmed, because the sound only condition was not 
implemented in this study. 



Effects of seductive messages on achievement 

It was found that there was no significant difference of recall test score and comprehension score in terms 
of the achievement among four different sources of seductive message conditions. The reason why these scores were 
not different can be explained by the “seductive details effecf ’. 

As for the recall test, previous research on the seductive details shows that adding interesting text that is 
irrelevant to the theme of a descriptive or narrative passage either reduces or doesn’t facilitate students’ 
remembering of the main idea in the passage (Gamer et al., 1992; Gamer et al., 1989; Midi & Baird, 1988; Wade & 
Adams, 1990). In this study, the mean score for text only and no message condition was 4.81 and 5.06 respectively. 
Although the result was not significantly different, the difference between mean scores shows that a seductive 
details effect occurred. 

Regarding comprehension test. Harp and Mayer (1997; 1998) found that students scored highest when 
only basic text was presented. Consistent with their study, the result of the comprehension study in this experiment 
also found that students who were in the no seductive message condition scored highest among the four message 
conditions although the difference was not statistically significant. 

However, it should be highlighted from the contrast analysis that students who were in both agent 
conditions scored higher than students in text only condition when the total number of keywords is considered. It 
means that students in both agent conditions actually recalled more keywords than the students in the text only 
condition. It can be inferred from this result that if the messages delivered by agents were not seductive in their 
content, the result might be different. It is because students actually recalled more information from an agent than 
they did from text although the messages were identical in all conditions. It emphasizes the importance of message 
design especially when pedagogical agents are embedded into computer based learning. 

Effects of seductive messages on attitude 

There were significant differences in relevance scores and confidence scores among the four message 
conditions. When seductive messages were presented, students perceived a higher relevance toward instructional 
material than did students in the no seductive message condition. However, students perceived higher confidence 
toward instructional material when only a companion role of agent was presented than the control condition where 
no seductive message was delivered. 

With respect to relevance, the reason students perceived higher score when seductive messages were 
presented then no seductive message was presented can be explained by the content of the seductive messages. 
Although the content of seductive messages was related to the main theme, it contained unimportant information to 
understand the main topic. However, students were exposed to a couple of examples that illustrated some stories and 
cases about intellectual property. According to Keller (1983, 2002), relevance is referred as value where personal 
worth and future value are immersed. Therefore, it is possible that students felt connected to the content of the 
seductive messages when seductive messages stimulated students’ perception of value by providing comment, 
anecdotes, or examples about intellectual property. 

Confidence scores were significantly different only between the companion role of agent and the control 
condition where no seductive messages were given. However, no significant difference was found between 
instructor role of agent and control condition. Confidence is defined as a positive expectation and support for 
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success by enhancing students’ belief in their competence (Keller, 1983, 2002). The fact that students recognized a 
companion agent and an instructor agent differently in this study implies that students felt more competence with a 
companion agent, because delivered messages and voice of the companion agent were consistent with students’ 
perception of an agent as a companion. However, it also should be noted that the difference on confidence scores 
between the companion agent and the instructor agent was minimal. 

Pedagogical agents, learning interest, achievement, and attitude 

In regard to the difference between the companion role of agent and the instructor role of agent, there were 
no significant differences in terms of learning interest, achievement, and attitude. This result is not consistent with 
those of earlier studies on the role of pedagogical agents which have found that agent perception and learning 
outcomes are different depending on what role of agent is implemented (Baylor, 2003). 

One possible reason why no significant difference was found between the roles of the pedagogical agent is 
that students did not recognize the agent persona of each agent. An agent has its character or personality based on 
students’ perceived persona combining image, animation, voice, and emotional expressions. In this study, although 
image, voice, and emotional expression were different between two agents, the messages delivered were identical 
each other. On the other hand, the study conducted by Baylor (2003) used different types of messages for each 
different role of agent. For example, in terms of messages, the expert agent provided information whereas the 
motivational agent provided encouragement and support, and the mentor agent provided both information and 
encouragement/support. 

Therefore, differentiating the delivered messages plays an important role for students to acknowledge the 
role of agent. However, identical seductive messages were employed to control the effect of seductive messages in 
this study. It led research to find no differences in regard to the learning interest, achievement, and attitude between 
the companion role of agent and the instructor role of agent. 

Conclusion 

Future research needs to move in four directions. First, as this study mainly focused on the effects of 
emotional interest, it is still necessary to consider the other component of situational interest that is cognitive interest. 
It has been suggested that cognitive interest adjuncts such as explanative summary, illustrations or messages 
influence students’ cognition by promoting structural understanding of the expository passage. And the attainment of 
structural understanding promoted a sense of positive affect about the passage. Therefore situational interest is 
affected by cognitive interest as well as emotional interest. This line of study will provide researchers with a big 
picture of how situational interest is promoted from two aspects, emotional interest and cognitive interest. 

Second, one of the assumptions in this study was that students will perceive the role of agents differently. 
Although no difference was found, many studies observed agent characteristics such as gender and age affect 
differently on the perception of students. Especially, the effects of messages were found to be very important in this 
study when they are delivered by agents. Therefore, future study is needed to examine how the cognitive interest 
messages or emotional interest messages affect students’ perceived situational interest when these messages are 
delivered by agents. 

Third, the contents of this study were mainly about intellectual property that was a type of expository 
information. Future study should extend the scope of material content to other types of text information such as math 
or science content where students actually need to apply what they have learned from the instructional material. Also, 
the narrative type of text needs to be tested. Previous research shows that the effects of seductive 
details/augmentation are different in opposite way depending on whether the context is expository information or 
narrative biography text. When a narrative biography text was used as a main text, none of previous studies showed 
the seductive-augmentation effect. In fact, the significant comparison showed that seductive augmentations helped 
learning of main ideas (Gamer & Gillingham, 1991; Schraw, 1998). 

Fourth, individual difference needs to be considered in future research. If students’ pre interest level is 
different at the beginning of instmction, the use of seductive details might be an effective strategy to improve 
students’ situational interest so that students will pay attention to the instmctional material. But if students are 
already highly interested in the instmctional material, there will be no need to improve students’ situational interest. 

Improving learning interest for successful learning is not a simple process. Many psychological constmcts 
are involved from an individual aspect and also a situational aspect. As an instmctional designer, therefore, it is 
necessary to approach the issue of learning interest from both of these aspects. 
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Introduction 

Although computers and other technologies have spread into every part of society, the integration of 
technology into our schools has been much less successful due to a number of barriers, both external (e.g., access to 
computers, software, and planning time) and internal (e.g., teachers’ preferred teaching methods, underlying beliefs 
about teaching and learning) to teachers (Ertmer, 1999). While access and training issues have been mitigated in 
recent years (Market Data Retrieval, 2002), teachers’ beliefs have only recently come to the fore. In order to 
promote favorable beliefs about classroom technology use, teacher educators are beginning to recognize the 
importance of engaging preservice teachers in authentic technology experiences (Albion & Ertmer, 2002; 

Windschhl, 2002). Recently, problem-based learning (PBL) has been highlighted as a potentially effective approach 
for changing preservice teachers’ beliefs (Levin, 2001; Tatto, 2003). By engaging preservice teachers in authentic 
problem solving involving collaborative decision-making, supported by technology, it is thought that PBL can serve 
as a catalyst for shifting teacher beliefs (Albion, 1999; Levin, 2001; Lundberg & Levin, 2003). 

Teachers’ Beliefs 

Labeled a “messy construct” by Pajares (1992), beliefs are still considered the “best indicators of the 
decisions individuals make throughout their lives” (p. 307). Kagan (1992) cited significant evidence supporting the 
relationship between teacher beliefs and their decisions about classroom practice. Specific to technology use, 
Niederhauser and Stoddart (2001) described patterns of use that were consistent with teachers’ personal beliefs 
about curriculum and instructional practice. Going one step further, other researchers (Becker, 1994; Berg, Benz, 
Lasley, & Raisch, 1998) have documented an association between low-level technology use and teacher-centered 
beliefs and practices, as well as between high-level uses and student-centered beliefs. Lor example, teachers who 
have student-centered beliefs tend to use multiple technology applications with problem-solving tasks (concept 
mapping software, Internet search engines) while teachers with teacher-centered beliefs tend to use single 
technology applications to present information (e.g., PowerPoint). It is becoming increasingly evident that if our 
future teachers are going to be prepared to use technology in ways that empower students, their pedagogical beliefs 
will need to shift from a traditional teacher-centered pedagogy to one that is more constructivist and student- 
centered (Lundberg & Levin; 2003; Richardson, 2003). 

Teachers’ Beliefs Regarding Technology Use 

Teachers’ beliefs have been emphasized previously especially in terms of teacher efficacy beliefs, 
expectations about students’ achievement, and students’ motivation (Bandura, 1997; Gibson & Dembo, 1984; 
Schunk, 1991; Woolfolk & Hoy, 1990). However, little research has been conducted focusing on teachers’ beliefs 
regarding technology use. Nonetheless, there is some research to suggest that there is a relationship between 
teachers’ technology uses and their pedagogical beliefs (Becker, 2001; Honey & Moeller, 1990). 

Parr (1999) and Tihema (2000) noted that encouraging teachers to develop new beliefs and stick with them 
is difficult. Even when teachers have a strong set of pedagogical beliefs, these ideas are often challenged and 
overruled by the conditions of the teaching environment. In other words, a pre- or inservice teacher who believes 
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strongly in student-centered learning may quickly adopt a teacher-centered approach when faced with an overloaded 
schedule, lack of administrative support, or other barriers in the classroom (Ertmer, Ross, & Gopalakrishnan, 2000). 
Although teachers agreed that technology makes their home, work, and school lives easier, technology integration is 
considered a difficult process (Parr, 1999; Tillema, 2000). Because computer technology is a relatively new 
phenomenon in the school environment, teachers still consider technology integration to be a complex process. 

According to Miller et al. (2003), teachers’ beliefs about technology are comprised of three related, but 
independent components: pedagogical beliefs about teaching and learning, self-efficacy beliefs about technology use 
(i.e., teachers’ confidence for using technology), and beliefs about the perceived value of computers for student 
learning. Russell, Bebell, O’Dwyer, and O’Connor (2004) found these three components to be the main predictors of 
teachers’ classroom technology uses. 

Current literature suggests that teachers’ beliefs can be changed through practices that emphasize reflection 
on personal beliefs, hands-on experiences, and engagement in authentic problems from K-12 classrooms (Copeland 
& Decker, 1996; Derry, Siegel, Stampen, & the STEP team, 2002; Tochon, 1999). For example, Derry et al. 
documented successful belief change after engaging preservice teachers in authentic displays of teaching (e.g., real 
cases, video cases). Through these authentic experiences, preservice teachers developed a better understanding of 
constructivist practices, became more effective at implementing constructivist methods, and achieved meaningful 
reflections on the nature of teaching and learning. Based on these results, problem-based learning (PBL) has been 
advocated as an effective approach for impacting teachers’ beliefs. 

Problem-Based Learning (PBL) 

A PBL learning approach is based on the use of ill-structured problem situations requiring students to 
develop expertise in information seeking and decision-making processes in order to solve the problems. Because the 
problem situations are messy, confusing, and complex, students need to gather information in order to understand, 
define, and solve the problems. During an authentic problem solving process, students are able to set their own goals 
and develop their own approaches. Under the guidance and coaching of a skillful teacher, students work 
collaboratively to inquire, investigate, and plan their learning activities (Torp & Sage, 1998). 

Impact of PBL in Teacher Education 

An increasing number of research studies demonstrate the value of PBL for enhancing students’ critical 
thinking skills, increasing motivation, improving problem solving skills, and facilitating social skills through group 
work (Duch, Groh, & Allen, 2001). As a catalyst for examining beliefs, PBL enables preservice teachers to 
recognize different perspectives and encourages them to articulate, defend, or change their current beliefs about 
classroom practice. That is, using problems and issues from K-12 classrooms, preservice students can connect their 
academic knowledge to issues in the field and thus examine their beliefs while engaging in problem solving, critical 
thinking, collaboration, and decision making (Lundberg & Levin, 2003). 

For example, Derry et al. (2002) found that preservice teachers gained more content knowledge in an 
educational psychology class with the PBL STEP (Secondary Teacher Education Project) approach than previous 
students taught in traditional ways. The STEP approach provided PBL activities and introduced various PBL video 
cases. Preservice teachers watched video cases of student interviews before and after instruction in the real K-12 
classroom and tried to understand why the students in the video case failed to learn. In the posttest, preservice 
teachers applied more appropriate concepts and described them in more sophisticated ways than on the pretest. 

Some studies have reported limited changes in preservice teachers’ beliefs after participating in their 
teacher education programs. Although the changes were limited, experiences with the cooperating teachers was 
found to be the most influential factor impacting their changes (Anderson, Smith, & Peasley, 2000; Borko & 
Mayfield, 1995). 

Therefore, to develop and change preservice teachers’ beliefs in teacher education, it is suggested that 
preservice teachers engage in problems and issues from K-12 classrooms, enabling them to connect their academic 
knowledge to their field experiences and then to examine their beliefs during the process (Richardson, 2003). 

The PBL approach using K-12 problems to contextualize important issues presents one method for 
changing preservice teachers’ beliefs. Although many educators have realized the importance of PBL, they have 
hesitated to implement PBL in their classrooms because of time, a lack of experience, and fear of assuming a 
facilitative role (Goodnough, 2003; Hemelo-Silver, 2004; Park, Ertmer, & Simons, 2005; Torp & Sage, 2002). 
However, PBL can provide an opportunity to change preservice teachers’ beliefs by motivating and engaging 
preservice teachers in reflection on their beliefs (Lundberg & Levin, 2003; Pierce, 1999; Pierce & Lange, 2001; 

Sage, 2000). 
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Pajares (1992) noted that it was hard to measure beliefs accurately. Therefore, Tatto and Coupland (2003) 
suggested that multiple data sources be used to measure teachers’ beliefs including surveys, observations of teaching 
practices, reflections, and interviews. Specifically for preservice teachers, they suggested including document 
analysis of instructional lesson plans across time. Because preservice teachers have limited classroom teaching 
opportunities, instructional lesson plans may offer a reasonable substitute for observing classroom teaching 
practices. 

Teachers’ teaching practices do not rely on impulse, but are directed toward certain preplanned outcomes 
(Putnam & Duffy, 1984). That is, lesson plans as a preplanned outcome, provide important information of a 
teacher’s intended teaching practices. For example, Kitsantas and Baylor (2001) examined teachers’ instructional 
lesson plans to determine their intended teaching practices. Peterson and Bond (2004) investigated differences in 
students’ instructional planning skills based on PBL approaches using different delivery methods (online vs. face-to- 
face) in teacher preparation. Previously, when instructional lesson plans were used (Kitsantas & Baylor, 2001; 
Peterson & Bond, 2004), the researchers evaluated the completeness of the plans rather than the theories underlying 
them such as behaviorism or constructivism. Furthermore, little research was found that analyzed instructional 
lesson plans as an indication of intended teaching practices associated with beliefs. In this study, I examined 
preservice teachers’ instructional lesson plans as an indication of their intended teaching practices, and as 
representative of their beliefs about teaching, learning, and technology. 

Although some literature is available regarding pre-service teachers’ beliefs, previous research has not 
investigated how to change beliefs regarding technology use. Specifically, little, if any, research has investigated 
how PBL might impact pre-service teachers’ beliefs regarding technology use (Hmelo- Silver, 2004; Tatto, 2003). 

The purpose of this study was to examine the impact of problem-based learning on teachers’ beliefs 
regarding technology use. Specifically, this study was guided by these primary research questions: 

1 . What is the impact of problem-based learning on preservice teachers’ beliefs regarding technology 
use? 

a) What is the impact of PBL on pre-service teachers’ pedagogical beliefs? 

b) What is the impact of PBL on pre-service teachers’ self-efficacy beliefs regarding technology 
use? 

c) What is the impact of PBL on pre-service teachers’ beliefs about the instructional value of 
computers? 

2. How do preservice teachers’ intended teaching practices change after participation in a PBL approach 
to technology integration? 

3. What are preservice teachers’ perceptions of the PBL experience? 

Methods 

This study used a mixed method approach to analyze the impact of PBL on preservice teachers’ beliefs 
regarding technology use. First, we measured changes in teachers’ beliefs regarding technology use. The teachers’ 
beliefs survey was composed of three components: teachers’ pedagogical beliefs, teachers’ self-efficacy beliefs 
about computer use, and teachers’ beliefs about the perceived value of computers for instructional purposes. 

Second, pre-and post-course lesson plans were used to measure changes in students’ intended teaching 
practices after participation in a technology integration course based on PBL. Participants’ intended teaching 
practices were analyzed with a lesson plan rubric. 

Third, course reflections and end of term interviews were used to investigate students’ perceptions of the 
PBL experience. Guided questionnaires were used for both reflections and interviews. 

Participants and Course 

Fifty-five preservice students, enrolled in four sections of a one-credit educational technology course, 
participated during spring 2005. This one-credit course was designed to help students achieve a greater 
understanding of issues and techniques related to integrating educational technology in K-12 settings through hands- 
on activities and discussion. The course met once a week, for two hours, over the first eight weeks of the semester. 

All the information provided by participants was kept confidential, and a unique coding system using 
section numbers and initials was used to track students’ pre- and post-survey data, pre-and post-course lesson plans, 
reflections and interviews. The institutional review board deemed the study exempt under human subjects 
guidelines. 
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Procedures 



The first class started with two digital video clips presenting interviews of school administrators including a 
middle school principal and a school superintendent. These video clips introduced the problem situation: the school 
district needed to hire competitive new teachers who could integrate technology into their classrooms. 

In the first week, students watched both digital video clips through the course website and then formed 
small groups of 3 or 4 students, according to their major area of study. Each group “created” a teacher candidate to 
apply for the new teacher position, including the development of a web-portfolio. 

In the second week, each student submitted a lesson plan, which included the use of technology, following 
specific guidelines that required them to describe learners, goals, assessment methods, resources, and so on. Early 
lesson plans were compared with lesson plans developed at the end of the course to examine how students’ 
intentions to use technology changed over the course. 

During the course, each group gathered additional information about technology integration and watched 
sample digital video cases of technology being integrated into K-12 classrooms. After watching the digital video 
cases, students discussed strengths and weaknesses of teachers’ technology practices. In addition, students submitted 
their reflections focusing on the PEL process at the end of weeks 3 and 8. 

As a capstone requirement, each group created a digital portfolio, using the artifacts created during the term 
to apply for the new teacher positions available in the school district. There were three main artifacts in the web 
portfolio: 1) an artifact to show the candidate’s skills (e.g., digital curriculum vitae) 2) an artifact to demonstrate the 
candidate’s knowledge (e.g., lesson plans integrating technology, video teaching demonstration), and 3) an artifact 
to show the candidate’s attitudes (e.g., an essay of teaching philosophy). Each group made 20-minute presentations 
to the middle school principal and the superintendent, as a way of applying for the available positions. 

Data Sources 

Data sources included a survey of teachers’ beliefs regarding technology use, lesson plans, students’ 
reflections, and interviews. The survey was completed at the beginning and at the end of the term. Students’ 
reflections were collected at the end of weeks 3 and 8. Interviews were conducted with five participants after the 
course ended. Each data source is described in more detail below. 

1) Teachers’ Beliefs regarding Technology Use Survey (TBTUS) 



In this study, teachers’ beliefs regarding technology use were measured by examining three aspects of 
teachers’ beliefs: teachers’ beliefs about student-centered learning, teachers’ efficacy for technology integration, and 
teachers’ perceived value for computers in teaching and learning (Chen, Bumam, Howie, Aten & Nambiar, 2003; 
Ertmer et ah, 2003; McCombs & Lauer, 1997; McKenzie, 2002). 

The Teachers’ Beliefs Regarding Technology Use Survey (TBTUS) contains 54 items including 35 items 
measuring teachers’ beliefs about student-centered learning, 7 items measuring teachers’ self-efficacy for 
technology integration, and 12 items measuring teachers’ perceived value for computers in teaching and learning 
(see Appendix A). Each component is described below. 

a) Teachers ’ beliefs about student-centered learning 

This survey was developed as a part of the Assessment of Learner-Centered Practice (ALCP) and has been 
tested over the past 4 years. One hundred fifty seven college instructors and 2,558 college students representing 12 
institutions and 10 states, participated in the field testing and initial validation (McCombs, 2002). 

The survey has 35 items (see Appendix A, items 1-35) divided into three sub-scales: Learner-centered 
beliefs about learners, learning, and teaching (14 items), non learner-centered beliefs about learners (9 items), and 
non learner-centered beliefs about learning and teaching (12 items). The Cronbach alpha reliability coefficient for 
each sub-scale ranged from .87 to 82. Learner-centered beliefs was .87, non learner-centered beliefs about learners 
was .83 and non learner-centered beliefs about learning and teaching was .82 (McCombs & Lauer, 1997). 

In this survey participants rate their levels of agreement on a 4-point Likert scale, from strongly disagree 
(1) to strongly agree (4) (e.g., “Students have more respect for teachers they see and relate to as real people, not just 
as teachers.” “Knowledge of the subject area is the most important part of being an effective teacher.” “I am 
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confident that I can use technology as an effective teaching tool.” “I am confident that I can meet the challenges of 
technology integration.” “My students can learn problem-solving more effectively with computers.” “I expect 
students to organize their thinking using Inspiration and other software program to make mind maps.”). 

A pilot study conducted fall 2003 (Park et al., 2005) showed a Cronbach alpha reliability coefficient for 
each sub-scale from .73 to .88. Learner-centered beliefs about teaching, learning, and learners was .88, non learner- 
centered beliefs about learners was .75, and non learner-centered beliefs about teaching was .73. Bai (2006) recently 
used this same survey with 230 students to examine teachers’ pedagogical beliefs. The results of her factor analysis 
indicated the same three factors as McCombs and Lauer (1997). 

b) Teachers ’ self-efficacy beliefs regarding technology use 

A recent study (Ertmer et al., 2003) used a seven-item survey. Teachers ’ Self-Efficacy for Technology 
Integration, to measure pre-service teachers’ self-efficacy for integrating technology (see Appendix A, items 36-42). 
Its reliability coefficient using Cronbach’ s alpha was .89, suggesting that the instrument was highly reliable. In 
addition, the items were evaluated by an expert in the area of self-efficacy and modified based on his suggestions, 
providing the instrument with content validity. The Cronbach alpha for this part of the teachers’ beliefs survey was 
.94 in the pilot study (Park et al., 2005). 

c) Teachers ’ beliefs about the perceived value of computers 

It is difficult to find a survey related to teachers’ perceived value of computers for teaching and learning, 
although, surveys related to teachers’ attitudes toward computers are common. Therefore, 12 items relevant to 
teachers’ perceived value for computers for instructional use were selected from the surveys employed by 
practitioners in the fields (Chen et al., 2003; McKenzie, 2002). The items were evaluated by an expert in the area of 
technology integration for pre-service education and modified based on her suggestions, providing the instrument 
with a measure of face validity. The Cronbach alpha for this part of the teachers’ beliefs survey was .88 in the pilot 
study (Park et al., 2005). 

2) Lesson Plans 

Participants turned in lesson plans both at the beginning of the course and at the end of the course and their 
lesson plans were graded according to a lesson plan rubric (see Appendix B). The rubric was composed of 7 
categories: 1) teachers’ roles, 2) students’ roles, 3) curricular characteristics, 4) learning goals, 5) types of activities, 
6) assessment strategies, and 7) types of technology. 

Pre-and post-course lesson plans (86 lesson plans) were analyzed by two independent graders using the 
lesson plan rubric and scored on each category according to a scale (where 1 = teacher-centered learning and 4 = 
student-centered learning). Scores for each category were summed to get a total score. The lowest score possible 
was 7 and the highest score possible was 28. The two graders graded the same 26 lesson plans (30% of all the lesson 
plans) and discussed discrepancies following guidelines of inter-rater reliability (Stemler, 2004). After the two 
graders reaching an agreement on using the rubric with these 26 lesson plans, each grader graded 30 lesson plans 
individually. The graders were blind as to whether the lesson plan was from the beginning or end of the course. 

They reached consensus with 94.86% agreement. The result of this analysis was examined to determine changes in 
preservice teachers’ intended teaching practices to use technology. 

3) Reflections and Interview 

To investigate preservice teachers’ perceptions of the PBL experience over the semester, students’ 
reflections were collected and five student interviews were conducted. Guided questionnaires were provided to 
collect students’ course reflections in weeks 3 and 8 (e.g., “What was impressive in the class?” “What do you feel 
could be improved in the class?”). A semi-structured interview questionnaire was used to interview students after the 
course (e.g., “Could you describe your experiences of group work during the course?” “Could you explain how you 
changed your beliefs about technology use after the course?”). Reflections were collected from all the students who 
enrolled in the course and five students volunteered to be interviewed. 

Results 
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Ql. What is the impact of problem-based learning on preservice teachers’ beliefs regarding technology use? 

Among 55 preservice teachers enrolled in the course, 43 preservice teachers completed both the pre- and 
post survey and lesson plans. The data from the Teachers ’ Beliefs regarding Technology Use Survey were used to 
determine the impact of PBL on teachers’ beliefs regarding technology use. A paired t-test {df= 42) indicated a 
significant increase in preservice teachers’ self-efficacy beliefs regarding technology use {t = 23\\p < .05) from 
pre- to post survey while there were no significant changes in preservice teachers’ pedagogical beliefs and 
preservice teachers’ beliefs about the instructional value of computers. 

Q2. How do preservice teachers’ intended teaching practices change after participation in a PBL approach to 
technology integration? 

Data from the pre- and post-course lesson plans were analyzed using a rubric developed for this purpose. A 
paired t-test {df= 42) showed a significant change in preservice teachers’ intended teaching practices {t = 2.14; p < 
.05) after participation in a PBL approach to technology integration. Preservice teachers’ scores increased from a 
pre-course lesson plan mean of 15.16 (5Z)=6.85) to a post-course mean of 17.16 {SD=1.\A). In general, preservice 
teachers tended to change their intended teaching practices from teacher-directed learning to student-centered 
learning. 

Specifically, the changes in the following two categories in the rubric significantly changed toward student- 
centered learning: “types of technology use” {t = 3.15; p < .01) and “curriculum characteristics” {t = 1.97; p =.05). 
For example, in the pre-course lesson plan, one student described how he/she would use PowerPoint and an LCD 
projector to deliver a lecture: “Because I seemed to learn well with this style in high school and college. . .1 believe 
lecturing in front of the class is the most useful method.” 

However, in the post-lesson plan, a greater variety of technology was used to help students’ learning. For 
example, some participants used technology to provide authentic examples through 3D models, hands-on 
experiences with simulation websites, and other useful resources for research. In addition, the number of activities 
involving group work and discussion increased, which does not necessarily translate into a student-centered 
approach but suggests that students were considering the use of activities that allowed the students to be more active. 

Q3. What are preservice teachers’ perceptions of the PBL experience? 

Preservice teachers’ reflections and interviews were analyzed to examine their perceptions about PBL 
experience. Although there were some suggestions for improving the course, reflections and interview comments 
were overwhelmingly positive. Preservice teachers liked having an authentic problem situation that helped them 
establish their own goals and provided opportunities to practice technology integration. Overall, preservice teachers 
perceived that the most effective aspects of the process included the group work, class discussions, and being able to 
contribute their own ideas. Preservice teachers enjoyed being able to share their ideas and to hear different 
perspectives. Following are a few representative statements from preservice teachers’ reflections and interviews: 

With the group work, I really liked that because I think you can learn from each other with problem-based 
learning. And if I didn’t have any ideas and I didn’t have an idea of answers, somebody else had ideas 
about it. It helped a lot. We could really help each other a lot. 

One thing that I found to work well was that the entire class was based mostly on discussions among 
students. Hearing other’s points of views opened your own to new things and we were able to feed off one 
another. 

This class wasn’t really like any of other ones I’ve taken since I’ve gotten to college. This class made me 
think on my own and use my own ideas to do major projects. 

Discussion and Implications 

The results of this study indicated that preservice teachers who participated in a PBL approach to learning 
about technology integration significantly increased their self-efficacy beliefs regarding technology use. In addition, 
the participants in this study significantly changed their intended teaching practices, specifically related to 
technology use, from teacher-directed to student-centered after participating in the PBL approach. Furthermore, 
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reflections and interviews supported the idea that preservice teachers benefited from the PBL approach and 
perceived positive impacts from group work, class discussion, and being able to provide input during the process. 

The survey results did not show any changes in pedagogical beliefs or in the perceived instructional value 
of computers from pre- to post-course. As many researchers have pointed out, preservice teachers’ beliefs are 
persistent and difficult to modify; preservice teachers bring a set of beliefs with them to teacher education programs, 
based on their own experiences as learners (Kennedy, 1997; Richardson, 2003). 

Nevertheless, some initial movement was shown in the way preservice teachers thought about their own 
teaching as represented by their lesson plans. This result supports findings by Derry and her colleagues (2002) that 
suggested that preservice teachers’ beliefs can be changed through authentic experiences and hands-on activities. It 
may be assumed that the hands-on activities with a variety of software and visible course products can affect the 
result. That is, preservice teachers could see distinctive changes and improvement of their skills, knowledge, and 
attitudes regarding technology integration through hands-on experiences and artifact development. In other words, it 
is easier to see the evidence of increased ability regarding technology skills through artifacts and it can affect self- 
efficacy beliefs regarding technology integration significantly. However, it is difficult to see evidence of changes in 
one’s pedagogical beliefs and beliefs about the instructional value of computers. 

The preservice teachers in this study perceived the PBL experience, particularly the use of group work and 
class discussion, to be a positive aspect of PBL. Pedersen and Liu (2003) suggested that collaboration is needed in 
the teacher education program and emphasized its potential impact on learning beyond the development of social 
skills. 

From an instructor’s perspective, a PBL approach may positively impact preservice teachers’ beliefs 
regarding technology use. Furthermore, this type of approach may help preservice teachers move their intended 
teaching practices toward student-centered learning as well as provide opportunities to practice skills, gain 
knowledge, and develop positive attitudes about technology integration. Although it is difficult to change preservice 
teachers’ beliefs radically through PBL, this study could be viewed as a beginning point to changing beliefs 
regarding technology use. Further investigation is planned, and the result of these investigations hopefully will help 
other teacher educators to develop and implement effective technology integration programs for preservice teachers. 
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Appendix A 

Teachers’ Beliefs Survey Regarding Technology Use 



Please read each of the following statements. Then decide the extent to which you agree or disagree. Circle the 
number to the right of the question that best matches your choice. Go with your first judgment and do not spend 
much time mulling over any one statement. PLEASE ANSWER EVERY QUESTION. 





Strongly 

Disagree 


Somewhat 

Disagree 


Somewhat 

Agree 


Strongly 

Agree 


1 


Students have more respect for teachers they see and can relate to 
as real people, not just as teachers. 


1 


2 


3 


4 


2 


There are some students whose personal lives are so dysfunctional 
that they simply do not have the capability to learn. 


1 


2 


3 


4 


3 


I can’t allow myself to make mistakes with my students. 


1 


2 


3 


4 


4 


Students achieve more in classes in which teachers encourage 
them to express their personal beliefs and feelings. 


1 


2 


3 


4 


5 


Too many students expect to be coddled in school. 


1 


2 


3 


4 


6 


If students are not doing well, they need to go back to the basics 
and do more drill and skill development. 


1 


2 


3 


4 


7 


In order to maximize learning, I need to help students feel 
comfortable in discussing their feelings and beliefs. 


1 


2 


3 


4 


8 


It’s impossible to work with students who refuse to learn. 


1 


2 


3 


4 


9 


No matter how bad a teacher feels, he or she has a responsibility 
not to let students know about those feelings. 


1 


2 


3 


4 


10 


Addressing students’ social, emotional, and physical needs is just 
as important to learning as meeting their intellectual needs. 


1 


2 


3 


4 


11 


Even with feedback, some students just can’t figure out their 
mistakes. 


1 


2 


3 


4 


12 


My most important job as a teacher is to help students meet well- 
established standards of what it takes to succeed. 


1 


2 


3 


4 


13 


Taking the time to create caring relationships with my students is 
the most important element for student achievement. 


1 


2 


3 


4 


14 


I can’t help feeling upset and inadequate when dealing with 
difficult students. 


1 


2 


3 


4 


15 


If I don’t prompt and provide direction for student questions, 
student won’t get the right answer. 


1 


2 


3 


4 


16 


Helping students understand how their beliefs about themselves 
influence learning is as important as working on their academic 
skills. 


1 


2 


3 


4 


17 


If s just too late to help some students. 


1 


2 


3 


4 


18 


Knowing my subject matter really well is the most important 
contribution I can make to student learning. 


1 


2 


3 


4 


19 


I can help students who are uninterested in learning get in touch 
with their natural motivation to learn. 


1 


2 


3 


4 


20 


No matter what I do or how hard I try, there are some students 
who are unreachable. 


1 


2 


3 


4 


21 


Knowledge of the subject area is the most important part of being 
an effective teacher. 


1 


2 


3 


4 


22 


Students will be more motivated to learn if teachers get to know 
them at a personal level. 


1 


2 


3 


4 
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Strongly 

Disagree 


Somewhat 

Disagree 


Somewhat 

Agree 


Strongly 

Agree 


23 


Innate ability is fairly fixed and some children just can’t learn as 
well as others. 


1 


2 


3 


4 


24 


One of the most important things I can teach students is how to 
follow rules and to do what is expected of them in the classroom. 


1 


2 


3 


4 


25 


When teachers are relaxed comfortable with themselves, they have 
access to a natural wisdom for dealing with even the most difficult 
classroom situations. 


1 


2 


3 


4 


26 


Teachers shouldn’t be expected to work with students who 
consistently cause problems in class. 


1 


2 


3 


4 


27 


Good teachers always know more than their students. 


1 


2 


3 


4 


28 


Being willing to share who I am as a person with my students 
facilitates learning more than being an authority figure. 


1 


2 


3 


4 


29 


I know best what students need to know and what’s important; 
students should take my word that something will be relevant to 
them. 


1 


2 


3 


4 


30 


My acceptance of myself as a person is more central to my 
classroom effectiveness of my teaching skills. 


1 


2 


3 


4 


31 


For effective learning to occur, I need to be in control of the 
direction of learning. 


1 


2 


3 


4 


32 


Accepting students where they are no matter what their behavior 
and academic performance-makes them more receptive to 
learning. 


1 


2 


3 


4 


33 


I am responsible for what students learn and how they learn. 


1 


2 


3 


4 


34 


Seeing things from the students’ point of view is the key to their 
good performance in school. 


1 


2 


3 


4 


35 


I believe that just listening to students in a caring way helps them 
solve their own problems. 


1 


2 


3 


4 


36 


I am confident that I can use technology as an effective teaching 
tool. 


1 


2 


3 


4 


37 


I am confident that I can use one computer effectively during large 
group instruction. 


1 


2 


3 


4 


38 


I am confident that I can develop effective lessons that incorporate 
technology. 


1 


2 


3 


4 


39 


I am confident that I can use technology effectively to teach 
content across the curriculum. 


1 


2 


3 


4 


40 


I am confident that I can overcome difficulties using technology in 
the classroom (time, scheduling, accountability). 


1 


2 


3 


4 


41 


I am confident that I can manage the grouping of students while 
using technology as a teaching tool. 


1 


2 


3 


4 


42 


I am confident that I can meet the challenges of technology 
integration. 


1 


2 


3 


4 


43 


Computers can provide instruction suited to individual students’ 
needs. 


1 


2 


3 


4 


44 


Computer use promotes student-centered learning and self- 
discovery. 


1 


2 


3 


4 


45 


Computers can enhance my students’ creativity and imagination. 


1 


2 


3 


4 


46 


Computers can engage my students in collaborative work. 


1 


2 


3 


4 
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Strongly 

Disagree 


Somewhat 

Disagree 


Somewhat 

Agree 


Strongly Agree 


47 


My students can learn problem-solving more effectively with 
computers. 


1 


2 


3 


4 


48 


Writing is easier for my students when they use computers. 


1 


2 


3 


4 


49 


I encourage and model smart choices about the tools students 
might use to accomplish tasks, using books, a spreadsheet or 
digital information when each one is the best. 


1 


2 


3 


4 


50 


I encourage students to use the Internet and e-mail to 
communicate with experts, other students and people from around 
the world to enrich their learning. 


1 


2 


3 


4 


51 


I expect students to organize their thinking using Inspiration and 
other software program to make mind maps. 


1 


2 


3 


4 


52 


I ask students to use networked computers to explore important 
questions and issues arising out of the content of my class. 


1 


2 


3 


4 


53 


I am making more time now than I used to for students to do more 
of the thinking, analyzing, interpreting, inferring, and synthesizing 
of information. 


1 


2 


3 


4 


54 


I am getting quite good at recognizing worthy uses of new 
technologies while avoiding the silly, trendy uses that waste time 
without delivering much of value. 


1 


2 


3 


4 
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Attrition In Distance Education: A Meta- Analysis 

Deborah A. Storrings 
Syracuse University 

Abstract 

This meta-analysis reviews twenty years of research on the effect of attrition predictor variables on distance 
education post-secondary learners. The predictor variables were derived from a series of attrition models, theories, 
and studies, and then further identified, defined, and grouped for additional analysis to determine their specific effect 
on attrition. Thirty studies on distance education attrition met the criteria for inclusion in this meta-analysis. There 
were seven moderator variables coded for each of 30 studies producing 284 separate (A:=284) effects. The overall 
effect size was 0.059, concluding there is no significant pattern in identifying variables that cause or contribute to 
distance education attrition within the empirical literature reviewed. 

Index words: Distance Education, Attrition, Completion, Persistence, Retention, Meta- Analysis, Effect Size 

Introduction 

Attrition has been a longstanding problem for almost all higher education institutions, and it is even more 
so with distance education seeing even higher rates (Carr, 2000; Holmberg, 1986; Parker, 2003). Carr states that 
attrition in distance education is not a recent discovery and while several research studies have addressed the 
subject, they have for the most part, all taken a different perspective on the issue. 

To date, there have been a number of wide-ranging studies that have been conducted to explore a variety of 
these predictor variables. For example, there are studies examining instructional design (Chyung, 2001; Van Schaik, 
Barker, & Beckstrand, 2003), demographics (Despain, 2003; Eisenberg & Dowsett, 1990; Frew & Weber, 1995), 
community engagement (Pugliese, 1994), program satisfaction (Schell & Thornton, 1985), and feedback and 
turnaround time (Taylor et al., 1986), to name just a few. There have also been a number of research studies that 
have tried to adopt many of the constructs outlined in Tinto’s (1975) retention model. These studies tried either to 
validate Tinto’s model, modify it, or create a new model or theory based on it, with varying levels of success 
(Bernard & Amunden, 1989; Fjortoft, 1995; Kember, Murphy, Siaw, & Yuen, 1991; Pugliese, 1994; Sweet, 1986). 
Finally, there are a number of studies that employed an experimental design to evaluate a single variable. For 
example. Frith and Kee (1994) used a control and treatment design to look specifically at the effect of differing 
communication methods, or Waschull (2001) who examined outcomes related to attrition based on mode of 
delivery. 

Understanding these studies and their numerous constructs is problematic since each study uses different 
sets of variables and different types of statistical analysis to predict attrition. Given these many different approaches 
to examining attrition in post-secondary distance education, it is not surprising that the findings in the research have 
been mixed at best. 

A review of the literature also indicated a lack of consensus on how many and what should be considered a 
valuable predictor variable. Kember (1989) notes, “The attrition process is undoubtedly a complex one, a theory 
that could fully explain every aspect of the attrition process would contain so many constructs that it would become 
unwieldy if not unmanageable” (p. 279). Therefore, this study conducted a meta-analysis to identify the overall 
effect of attrition in distance education, and to investigate possible sources of variation among the predictor 
variables that may explain the diverse findings within the literature on distance education attrition. This quantitative 
synthesis investigated following questions: 

1 . What is the average effect size over distance education attrition studies, and is 
it different from zero? 

2. Is the effect size related to study characteristics, and if so, do they have a 
positive or negative affect on attrition in distance education? 

3. Were effect sizes influenced differentially by methodological characteristics? 

4. Were effect sizes influenced differentially by substantive characteristics? 
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To answer these questions, this study synthesized peer-reviewed thirty research studies on attrition in 
distance education. The study employed the research method of meta-analysis based on the works of Glass, McGaw 
and Smith (1981), Orwin (1983), Hedges and Olkin (1985), Mullen (1989), Rosenthal (1979; 1991), and Lipsey and 
Wilson (2001). 



Review of Literature 

The review of the literature began with an examination 20 years of quantitative studies to analyze the 
phenomenon of attrition in post-secondary education, specifically distance education. The review started with 
Vincent Tinto’s 1975 higher education predictive retention model, which is known to be one of the most extensively 
cited and the most widely tested models to date (Kember, 1989). Tinto’s model was designed to address traditional 
four-year university students in a face-to-face arena. However, this model has also been cited in numerous studies 
(Kember, 1989; Sweet, 1986; Smith, 2001; Taylor et al., 1986) related to distance education and has been used to 
launch many new and diverse attrition theories, models, and studies. Tinto’s model concluded that a student’s 
commitment to college, and his or her personal academic goals and background, have a direct impact on their 
persistence. Tinto claims that based on how well a student integrates into the scholastic and social systems of the 
institution will be determined whether or not they dropout. In 1983, Pascarella and Chapman’s research on attrition 
also concluded that a student’s interaction with college life was connected to retention, further supporting Tinto’s 
claim (Halpin, 1990; Pascarella & Chapman, 1983; Pascarella & Terenzini, 1998; Terenzini & Pascarella; 1980). 

In later years, Tinto realized the narrow scope of his model and in his 1982 article entitled, “Limits of 
theory and practice in student attrition”, he stated that if his earlier model were to be used for nontraditional forms of 
education such as distance education, then modification would be required. David Kember (1989) did just that by 
taking Tinto’s model and redesigning it for use with distance education students. Kember illustrated that distance 
education students were most often part-time and older than those who studied on campus, and for this reason 
factors other than college life and academic integration weighed more prominently in their decision to drop out. In 
his research on attrition, Kember developed the Distance Education Student Progress inventory (DESP). With this 
instrument, he concluded that a number of external variables such as employment and family obligations played a 
strong role in the attrition of distance education students. Further, Kember pointed out that looking at single 
variables is of little value and argued that the examination, use of, and integration of multiple variables would 
enhance the successful prediction of attrition. Kember’ s continued research in 1990 concluded that intrinsic 
motivation was a strong predictor of retention, especially when students perceived their educational experience was 
significant to their job or personal interests. 

Kember’s research was confirmed when Fjortoft (1995) experimented with a model designed to predict 
adult student attrition in distance education. Fjortoft concluded an adult student was less likely to drop out if he or 
she expected to gain “tangible career related results” (Fjortoft, 1995, p. 2). Fjortoft identified five variables that 
could be used to predict attrition among adult learners in a professional distance education program; age, previous 
college education, intrinsic job satisfaction, attitude towards learning, and perceived benefits of completion. 

An alternative model researching the nontraditional student, still based on Tinto’s framework, was Bean’s 
1980 Causal Model of Student Attrition. This model investigated the “student’s institutional fif ’ (Rovai, 2003, p.3) 
by using the student’s personal characteristics (demographics) and his or her academic backgrounds as variables for 
determination. Bean argued that students dropped out for the same reasons that employees left their jobs, including 
factors such as relocation and better opportunity. In 1985, Bean worked with Metzner on a new, more refined model 
for nontraditional students, which defined four variable sets that could predict attrition: background variables, 
environmental variables, academic variables, and intent to leave (Bean & Metzner, 1985). Bean and Metzner’s 
model was one of the first to argue that external variables such as hours of employment, family obligations, and 
personal finances played a major role in predicting attrition for nontraditional students. 

In 1988, Bean’s work was modified by Billings who developed an attrition model centered around four 
variable sets to predict attrition. Two of the four variable sets closely paralleled Bean’s and Tinto’s academic 
performance and environment variables. Billings argued that the other two variable sets that predict attrition for 
student parting in correspondence course are supporting structure and course variables (Billings, 1988). 

Each study reviewed has its own constructs similar to Billings, sometimes building off someone else’s 
variable(s), sometimes redefining those same variable(s), and other times just manufacturing their own. For 
example. Sweet (1986) found that perceived helpfulness of faculty and non-course discussions with instructors were 
variables that could predict attrition accurately. Pugliese (1994) found that psychological factors such as loneliness, 
communication apprehension, and locus of control were not valid predictor variables of attrition for distance 
education students. Pugliese did conclude that distance education “appears to be a social equalizer when it comes to 
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receiving course credits” (p. 34). Taylor et al. (1986) found that quick turnaround time on correspondence and 
review of assignments could be positively correlated to retention. Contrary to several models that suggest past 
education is a significant attrition predictor, Chacon-Duque (1985) concluded that past academic experience and age 
were not valid predictors, but variables such as quality of course materials, variety of media, and planned student 
support were exceedingly significant indicators. 

From the literature, two distinctions of the status of research on student attrition in distance education were 
evident. First, there seems to be no variable set(s) that is defined in the same way, nor are there any theories or 
models that appear to rise above the rest. Tinto’s model does emerge as a basic building block for most other 
theories and models, including those developed for nontraditional students and for distance education. However, 
each researcher tends to interpret or modify Tinto’s variable sets in his or her own manner, while others continue to 
create new and more encompassing variable sets, hence increasing the complexity of attrition predictor variables. 

Second, the research seems to be going in many different directions simultaneously while also producing a 
high number of contradictory reports. There were for example a number of studies that validate the significance of a 
preceding study but then argue that the past findings need to be altered by either deleting a variable set(s) or by 
including a new or modified variable set(s). Conversely, immediately after a research study identifies a new 
variable set(s), it is directly followed-up by another study suggesting an opposite conclusion. This is partially 
because many studies look at unique populations or use a different way in which to define and calculate attrition. In 
view of all these variations in the literature on this topic, it would currently be difficult to decide on any type of 
implementation strategy to prevent attrition in distance education. 

Method 



Inclusion Criteria 

All studies used in this meta-analysis had to meet the following inclusion criteria throughout all stages of the 
literature search: 

1 . Studies must incorporate attrition rates or data for post-secondary students partaking in distance education. 

2. Studies must have sufficient descriptions of population characteristics. 

3. Studies need to have been published in an English-language peer-reviewed journal or English-language 
dissertations, ensuring that the quality of the study meets research standards in the field. 

4. Studies published before 1984 will not be included in order to provide for a more contemporary database. 

5. Studies must contain statistical documentation to allow for calculation or estimation of effect size. 

Literature Search Methods 

The first round of searching consisted of a systematic scan of the following databases: Education Resources 
Information Center (ERIC), PsycINFO, Education Abstracts, Social Science Abstracts, and Dissertation Abstracts. 
The computer search strategy used the following keyword search terms: attrition, dropout, non-completer, non- 
persister, withdrawal, distance, online, distributive, web-based, computer based, and correspondence education and 
training. This preliminary round of database searches produced 104 articles and 75 dissertation abstracts. Of these 
only 27 articles and 61 dissertations meet all of the inclusion criteria. Many of those eliminated were qualitative 
studies or case studies that included no statistical information. A number of others were eliminated because they 
met most but not all of the inclusion criteria. For instance, some researched secondary school attrition or others used 
a single group non-experimental design. 

Most of the remaining 89 candidates only survived because their abstracts were not specific enough to 
exclude them without further investigation. After gathering full copies of each study, the field was further reduced 
to nine articles and twenty dissertations. A second round of inquiry was performed by searching individual journals 
that were identified through the primary search and finally an ancestral search was completed. These additional 
searches produced three more dissertations for a total of nine articles and twenty-one dissertations. 

Coding 



Studies were coded into moderator variables that fell into two categories, substantive and methodological. 
There were four substantive variables; study focus, institution type, student type, and attrition factors and three 
methodological variables: statistics reported, article type, and publication year. Each is these was further broken 
down as outlined in Table 1. 
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Table 1. - Coding Scheme 




Substantive 


Score 


Institution Type 


Traditional 4 year 


1.1 


Community/Junior/Tech College 


1.2 


Open University 


1.3 


Corporate Training and Continuing Education 


1.4 


Student Type 


Traditional 18-24 


2.1 


Graduate/Post Bachelors 


2.2 


Professional or Life Long Learner 


2.3 


Study Locus 


Course Attrition 


3.1 


Program Attrition 


3.2 


Attrition Lactors 


Demographic 


4.1 


External 


4.2 


Internal 


4.3 


Mixed 


4.4 




Methodological 


Publication Year 


1984-1994 


5.1 


1995-2004 


5.2 


Source Type 


Peer-reviewed Journal 


6.1 


Doctoral Dissertation 


6.2 


Research Method 


Mean and Standard Deviations 


7.1 


F,t,r,X^ 


7.2 


Odds ratio\2x2 tables 


7.3 


Mixed 


7.4 



Before beginning the meta-analysis statistical procedure, an inter-rater reliability analysis was performed 
on the coding scheme. The coding was performed by two independent researchers returning a Kappa score of .6825, 
falling into the substantial range. 



Results 



Characteristics of the Studies 

Thirty studies on distance education attrition met the criteria for inclusion in this meta-analysis. Twenty- 
one were dissertations and nine were from peer-reviewed journals. The studies were published from 1989 to 2003, 
with the median year being 2000. Thirty-nine percent of the studies (^=12) fell into the early years coding category, 
those published from 1984 to 1994; the remaining 61 percent (^=18) were coded as later years, from 1995 to 2003. 
Sample sizes ranged from a low of 38 to a high of 1504 for an approximate 9769 participants, with a median of 252 
participants per study. The majority of studies, 50 percent (^=15), examined distance education attrition at 
traditional four-year institutions; 27 percent (8) at community colleges; 17 percent (^=5) at open universities; and 
the remaining researched professional development and/or corporate training. Thirty-seven percent of the studies 
explored only internal attrition variables {n=\ 1), including such items as assignment turnaround time, course 
materials, and amounts of faculty interaction. Twenty-seven percent analyzed only demographic attrition variables 
(8) such as age, past education, distance from the offering institution. The remaining 36 percent researched 
variables from all three variable sets (mixed), demographics, internal and external, while no studies focused 
exclusively on external attrition factors. 
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Publication bias, the file drawer, and normality 

Publication bias was addressed by creating a funnel plot of effect size against their corresponding sample 
size (Light & Pillemer, 1984) as seen in Figure 1. There were two types of studies used in this meta-analysis, those 
selected from peer-reviewed published journals, and those from doctoral dissertations. Since the effect size 
calculated for both the published journal studies and the dissertations are relatively the same (journals d= 0.0609, 
dissertations d= 0.0569), it seems evident that no significant journal bias is present. 

Figure 1 - Funnel Plot 




To look further for publication bias and to assess normality, a normal quartile plot was created (Figure 2). 
All of the studies fall within 95 percent confidence interval with the exception of the Hansen dissertation, which had 
an effect size that was medium to large (.6299), while also having an extremely small variance (.0023). The Hansen 
study appears to be out of the normal population, which is consistent with the overall effect size results of d=.059. 

Figure 2 - Normal Quartile Plot 
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Overall Effect size 



Two models are generally used in meta-analysis; the fixed effects model and the random effects model. The 
fixed effects model assumes that all of the variability between effect sizes is due to sampling error or the luck of the 
draw. The random effects model suggests that the variability between effect sizes is due to sampling error plus 
differences in the population of effects. For example, other random differences between studies may include 
differences in procedures, settings, methods, and population grouping, all of which go beyond simple subject level 
sampling error (Lipsey & Wilson, 2001). “Thus the observed variability in sample estimates of effect size is partly 
due to the variability in the underlying population parameters and partly due to the sampling error of the estimator 
about the parameter value” (Hedges & Oklin, 1985, p. 191). A more obvious difference between the two models is 
that the fixed effects model tends in general to show larger confidence intervals as well as a more significant overall 
effect size, presenting a less conservative approach. Although all of the studies had distance education attrition as 
their dependent variable, there were more than one hundred different attrition factors, several diverse and unique 
treatments, and varying types of subject classification and grouping. More importantly, there were several different 
statistical methods used to calculate effect sizes between and within studies, therefore a random effects model 
approach was deemed appropriate (Rosenberg et al., 2000). 

Using the random effects model. Table 2 shows the results of the meta-analysis. From the 30 studies there 
were 284 effects calculated (A:=284). The median for A: was four, with a maximum of fifty-eight and a minimum of 
one. The overall effect size of d is 0.059, which is considered to be a non-significant result as defined by Cohen 
(1988) and Fipsey (1990). Although the effect size is different from zero, it is not near enough to the .20 that would 
be considered a small but significant effect size. 

Fooking closer at the results in Table 2, the lowest effect size appears in the Cordover dissertation with - 
.3151, and the highest is the Thompson dissertation at 1.1206, with an overall median effect size of .062, which is 
extremely close to the overall study J of .059. There were eighteen studies showing a positive effect size indicating 
that the independent variable (demographic, external or internal) could indicate changes in or predict the dependent 
variable (distance education attrition), while the remaining twelve studies showed the opposite. The overall effect 
size was positive at d= 0.0590 with a variance of 0.0808 having a 95 percent confidence level between -0.0397 and 
0.1577. A random (pseudo) resampling (999 times) was done to produce a bootstrapped 95% confidence interval of 
-0.0337 to 0.1517, which is extremely close to the sample’s 95% confidence interval, again reinforcing the 
normality of the data. A homogeneity analysis produced a g of 30.1341, df 29, with p equal to 0.4073. Since Q was 
not statistically significant at the .05 level, it can be logically assumed that the data in the sample sufficiently 
represent the population (homogenous). 

The normality of the sample, its homogeneity, and the result of <7= 0.059 gives empirical evidence that 
there are no attributable variables to identify distance education students who complete and those who attrite. There 
appears to be no clear pattern in identifying, classifying, or accounting for distance education attrition within the 
distance education empirical literature. This meta-analysis posed four primary questions. The first asked, “Is there 
an average effect size across the 30 studies and is it greater than zero?” The answer is yes, there is an effect size 
greater than zero, but it is not sizeable enough to be considered a significant finding, hence eliminating the need to 
address the next three research questions. 

T able 2. - Random Model Meta-analysis results 

Effect 95% Confidence 



Study 


Year 


n 


Size 


Var 


k 


Fewer 


Upper 


Atack 


2001 


39 


0.7233 


0.8130 


4 


-0.8701 


2.3167 


Beaty 


1994 


103 


0.0522 


0.0100 


1 


0.0326 


0.0718 


Bernard & Amunden 


1989 


361 


-0.1337 


0.0249 


16 


-0.1825 


-0.0849 


Bourke 


1997 


239 


-0.2561 


0.1958 


2 


-0.6399 


0.1277 


Caldwell 


2003 


126 


0.1593 


0.0081 


2 


0.1434 


0.1752 


Cordover 


1996 


89 


-0.3151 


0.0116 


1 


-0.3378 


-0.2924 


Crabtree 


2000 


1504 


0.1295 


0.0007 


1 


0.1281 


0.1309 


Diaz 


2000 


91* 


-0.0932 


0.0665 


15 


-0.2235 


0.0371 


DiBisceglie 


2002 


444 


0.2271 


0.0023 


8 


0.2226 


0.2316 


Dille & Mezack 


1991 


151 


-0.1882 


0.0335 


3 


-0.2539 


-0.1225 


Frith & Kee 


1994 


174 


0.2230 


0.0932 


1 


0.0403 


0.4057 
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Foley 


1993 


684 


0.0294 


0.0051 


2 


0.0194 


0.0394 


Friedman 


2003 


75* 


-0.1383 


0.0773 


7 


-0.2898 


0.0132 


Glickman 


2003 


141 


0.0887 


0.0506 


49 


-0.0105 


0.1879 


Hansen 


2000 


435 


0.6299 


0.0023 


1 


0.6254 


0.6344 


Langenback & 
Korhonen 


1988 


400* 


0.1675 


0.0084 


9 


0.1510 


0.1840 


Loo 


2003 


38 


0.4429 


0.1708 


2 


0.1081 


0.7777 


Moore 


2001 


249 


0.1196 


0.0041 


1 


0.1116 


0.1276 


Osborn 


2001 


396 


-0.2536 


0.0294 


15 


-0.3112 


-0.1960 


Osorio 


1987 


187 


-0.0303 


0.0054 


18 


-0.0409 


-0.0197 


Ostman & Wagner 


1987 


676 


0.4487 


0.0360 


4 


0.3781 


0.5193 


Parker 


1994 


177 


-0.6373 


0.1167 


1 


-0.8660 


-0.4086 


Reamer 


1990 


298* 


0.0726 


0.0038 


19 


0.0652 


0.0800 


Schwartz 


1999 


302 


0.0137 


0.0033 


1 


0.0072 


0.0202 


Taylor et al 


1986 


674* 


-0.0549 


0.0998 


12 


-0.2505 


0.1407 


Tello 


2002 


669* 


-0.6208 


0.1574 


5 


-0.9293 


-0.3123 


Thompson 


1999 


258* 


1.1206 


0.3360 


7 


0.4621 


1.7791 


Tsay 


1999 


589* 


-0.2836 


0.0095 


12 


-0.3022 


-0.2650 


Valentine 


1986 


256 


0.2515 


0.0040 


58 


0.2437 


0.2593 


VanSchaik & Baker 


2003 


119 


0.1078 


0.0436 


4 


0.0223 


0.1933 


Overall 






0.0590 


0.0808 


284 


-0.0397 


0.1577 



* - Approximate n values 



Discussion 

The present synthesis summarizes research examining the relationship between distance education 
completers and non-completers. As seen throughout the literature review, there are many models, theories, and 
studies that suggest a specific variable or sets of variables that can account for or predict attrition in post-secondary 
distance education, but the outcome of this meta-analysis suggests the opposite. The overall result of an effect size 
of .059 indicates that there are no factor(s) strong enough to identify significant differences between distance 
education completers and non-completers. The result of this study supports Rovai’s conclusion that, “There is no 
simple formula that ensures student persistence” (Rovai, 2003, p.l2). Further, indifferent to the claims of a number 
of other researchers, this study suggests there is no single variable or set of variables that can clearly be attributed to 
the prediction of distance education attrition. 

An important finding of the study that warrants discussion is the quality of the studies in the distance 
education literature, since this is of prime concern for all meta-analyses. It would have been beneficial to 
incorporate a greater number of studies in this meta-analysis, but unfortunately, there were a large majority of 
studies in the research base that were methodologically flawed in one way or another. As a whole, the studies 
seemed to lack methodological rigor, with many studies using non-equivalent comparison groups rather than true 
random assignment. Some studies were flawed by simple errors such as number reversal in tables or text, groups 
transposed, or contingency tables missing strategic information. There were also several studies where the n was 
reported inconsistently within the body of the article with no given explanation. Due to the poor quality of research 
in this area, a sizeable number of studies that initially appeared to meet the selection criteria were eventually 
eliminated. 

This study ‘s findings support Phipps and Merisotis (1999) findings that were prepared for the American 
Federation of Teachers and the National Education Association suggesting “there is striking evidence of the fact that 
there is relative paucity of true, original research dedicated to explain or predict phenomena related to distance 
education” (p.l3). Further, they suggested, “merely being published in a journal or a book, for example, does not 
guarantee the quality of the study or that it was reported accurately” (p.l9). This meta-analysis can confirm there 
was no difference between peer-reviewed journal articles and doctoral dissertations, both producing low but 
equivalent effect sizes {d=.Q6 and J=.057, respectively). This suggests that there may be no difference in quality, at 
least between these two types of research reports. 

Further, there was no difference between the two sets of publication years, 1984-1994, and 1995-2004. 

The synthesis of research covered a twenty-year span in which the delivery mechanisms for distance education have 
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changed considerably, as have the numbers and the characteristics of today’s distance learners. The speed at which 
educational technology is changing and the ever-increasing percentages of the population partaking in such, leads to 
the view that the results of this synthesis may be somewhat historical and somewhat less generalizable in the 
distance education arena of today. Despite these obvious differences in technology for service delivery, this study 
showed only a minute difference between those studies originating between 1984 to 1994 (^=12, J=.067) and those 
from 1995 to 2004 (^=18, J=.059). This would indicate that technology does not play a significant role in distance 
education attrition, that it lies in other areas. 

Research Implications 

As identified in the findings of this synthesis, there is a need for improved research methods in distance 
education research. Phipps and Merisotis (1999) rejected the findings in Thomas Russell’s 1999 The No Significant 
Difference Phenomenon because of the poor quality of distance education research. They criticized many of the 
articles, especially those pertaining to original research, for not controlling for extraneous variables, not using 
random assignment, and using questionable instruments that lacked validity and reliability for data collection. This 
meta-analysis seems to validate those criticisms, as seen by the small number of usable studies. Of the 88 studies 
that initially appeared to meet the selection criteria from the primary search, only 3 1 percent (n=21) survived the 
screening process. 

The quality of educational research has recently become a hot topic among legislators as well as academics. 
Recently enacted federal legislation, including the No Child Left Behind Act (2001) and the Education Science 
Reform Act (2002) has raised demands for “scientifically based” educational research. There is an intense debate on 
what constitutes a scientific base for educational research (Eisenhart & Towne, 2003), but its ultimate goal seems to 
be to produce more accurate studies based on methodologically sound research design, solid measurement, and 
appropriate statistical analyses. 

Another compelling implication of this meta-analysis is that it supports Kember’s (1990) notion that 
“dropping-out from distance education is a complex phenomenon” (p. 11). The true complexity with attrition is 
visible in how each study used in this synthesis chose to define or not to define the term attrition itself Ten of the 
studies that were included gave no clear definition of attrition while others made in-depth statements. Their 
descriptions, explanations, and classifications were extremely diverse in nature. For instance. Loo (2003) used a 
somewhat detailed explanation of attrition; “students who have registered in an online course, but did not meet the 
course requirements of a passing grade, and did not withdraw by the official drop date or received an “F” or “IF” for 
incomplete fail were considered drop outs” (p. 27). Conversely, Dille and Mezack (1991) merely stated that attrition 
is the number of withdrawals from the course during the semester. An even more simplistic definition came from 
Ostman and Wagner (1987) who narrated, “respondents decided for themselves whether or not they were dropouts” 
(p. 53). Clearly, these researchers have different conceptual ideas on how attrition should be defined and 
operationalized. Accordingly, a given student could be labeled a dropout in one study, and as a persister in another. 
These circumstances make it enormously difficult to find meaning in what is in many other ways a particularly 
diverse literature. It appears that achieving consensus regarding the definition to the term attrition would help 
clarify some of the obscurity found among and between attrition studies. As far back as 1978, Pantages and 
Creedon noted, “ideally, studies of attrition should incorporate... a definitive unambiguous operational definition of 
dropouf ’ (p.56). This is not yet the case, since studies produced in the past five years still have the limitation of not 
producing or using a single clear definition of attrition. 

Recommendations 

The most noteworthy recommendation as a result of this study is the need for longitudinal studies. Almost 
all of the studies reviewed were a quick snapshot in time, with the largest percentage of studies researching attrition 
over only one or two semesters. This again goes back to the quality of research in the area. It would be expected 
that longitudinal studies might be missing from the doctoral dissertation group, but more should be found in articles 
published in peer-reviewed journals. The need for longitudinal studies is evident in that all educational research is 
unique and each institution, classroom, or student may vary significantly from place to place, hence the 
accumulation of research evidence over time and across studies (meta-analysis) may be the best way to produce 
consistent and accurate findings (Waxman, Lin & Micko, 2003). 

There are still many unanswered questions about distance education attrition, but research can play a key 
role in finding answers and solutions. The result of this research recommends not only the need for an accepted 
definition of distance education attrition, but the need for research studies that employ better measurement tools. 
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more pre-tested data collection instruments, and more detailed statistical analysis. Research studies in education 
need to focus on better design and more control for extraneous variables, so that the justification for study findings 
is stronger and defendable. 
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Abstract 

In the current study, researchers examined learner experiences and preferences for a number of 
instructional activities that promote learner-instructor, learner-learner, and learner-self interactions in online learning 
environment. Statistical procedures were used to determine the discrepancies between learner experiences and 
preferences for these instructional activities. The follow up interviews further explained why certain instructional 
activities were less preferred while they were frequently used in practice. Results of this study have meaningful 
contribution to current practice of distance education. 

Introduction 

Although literature shows the importance of interactions for quality distance education, interaction seems 
lacking in many online courses (El-Tigi & Branch, 1997; McGorry, 2002). Instructor unfamiliarity with technology 
seems to be one of the key reasons why they do not know how to enhance online interactions in practice (Stenhoff et 
ah, 2001). A study from the National Center for Education Statistics (2001) concurs on this point. It reports that one 
of the greatest challenges of teaching online is the lack of guidance on how to teach with new technologies. This is 
often because the skills required for teaching online are quite different from skills required for face-to-face teaching 
(Daniel, 2003; Shutt, 2003). Therefore, the fundamental issues of teaching online are not limited to the issues of 
technology; actually they are pedagogical in nature (King & Doerfertl, 1996; Oswald, 2003). It seems obvious that 
online instructors should re-examine the original assumptions and teaching methods in order to provide rich learning 
experiences to online learners. As part of this concern about helping online instructors increase course quality and 
given the fact that there is lack of research on the pedagogical issues of online interactions (Kearsley, 1995; Jiang & 
Ting, 1999), the current study is conducted to investigate how interactions are promoted in practice and whether 
they align well with student preferences for online interactions. 

Research Questions 

Specific research questions of the study are: 

• Q 1 : Which instructional activities are used to promote learner-instructor, learner-learner, and learner-self 
interactions in online learning environment? 

• Q2: What are the learner preferences for these instructional activities? Are there discrepancies between 
student experiences and preferences of these activities? 

• Q3 : Why do learners prefer certain instructional activities over others? How we can change current practice 
to make online learning a more pleasant experience for learners? 

Methodology 

This study was conducted in a large mid western university. The survey was distributed to a total of 463 
online MBA students and alumni. A total number of 188 students responded and the response rate was 41%. Survey 
respondents who were willing to participate in the follow-up interview were asked to provide an e-mail address 
when filling out the survey. Among these volunteers, 1 1 students were selected for a 45 -minute one-on-one 
telephone interview. 
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Results 



Results showed that online interactions happen primarily through team-based collaborative tasks, prompt 
feedback from instructors, asynchronous lectures, asynchronous discussions, sharing resources with peers, and 
regular office hour consultations. Anything requiring synchronous involvement or taking role in class activity is not 
used much currently in online learning. Learner self-reflection on the subject matter happens mostly through reading 
discussions, using what has been learned in real life situations, and in responding to questions independently. Online 
learners don’t usually “come to class” with questions prepared. 

As shown in the Table 1.1, the rank order of the experiences and preferences are fairly consistent with each 
other except the “check on student progress regularly.” Instructor checking on student progress regularly was not 
perceived as a frequently occurring activity, but it was preferred higher (ranked 3rd) on the list. This indicates that 
students want to be checked on their academic progress regularly by instructors. 

Table 1.1: Relative ranking differences of experiences and preferences for 
learner-instructor interactive activities 



Instructional activities for learner-instructor interaction 


Ranking 

Experiences 


Ranking 

Preferences 


Prompt feedback 


1 


1 


Asynchronous lectures 


2 


2 


Instructor participate in class-level discussions 


3 


3 


Regular office hours for consultation 


4 


5 


Grade on student responses to discussion questions 


5 


7 


Check on student progress regularly 


6 


3 


Instructor participate in group-level team discussions 


7 


6 


Social communication with students 


8 


6 


Synchronous lectures 


9 


9 



Among all the leamer-leamer interactive activities, “having class-level large group asynchronous 
discussions” had the largest ranking difference between its experiences and preferences order on the list. The 
reasons for this relatively huge rank order difference was addressed in the follow up interviews. Although it was 
used often in practice, student did not prefer to engage in class-level large group discussions. Follow up interviewed 
revealed that the reasons were mainly because of the large class size, duplicated ideas, and unclear rules and 
expectations. 



Table 1.2: Relative ranking differences of experiences and preferences for 
learner-learner interactive activities 



Instructional activities for learner-learner interaction 


Ranking 

Experiences 


Ranking 

Preferences 


Have team-based collaborative projects and assignments 


1 


2 


Have small-group asynchronous discussions among students 


2 


4 


Share course-related information and resources with my peers 


3 


1 


Have class-level large-group asynchronous discussions 


4 


8 


Have informal chats with peers where we share experiences and 
beliefs etc 


5 


3 
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Have synchronous chat sessions for students to engage in ’’real-time” 
discussions 


6 


7 


Give feedback to my peers 


7 


6 


Get feedback from my peers 


8 


5 


Utilize task roles 


9 


11 


Have student online debating team activities 


10 


10 


Have online role-playing activities (e.g., playing different roles in a 
case). 


11 


9 



The relative ranking differences of experiences and preferences in Table 1.3 illustrate that “reading over 
each other’s discussions in the online forums” was the most frequently used learner-self interaction activity; but it 
was not the most preferred item on the list (ranked fifth). The reasons for low preferences toward this activity were 
similar to the reasons for low preferences toward the class level large group discussions. These low preferences were 
mainly due to the large class size, duplicated ideas, and lack of instructor participations in discussions. 

Table 1.3: Relative ranking differences of experiences and preferences for 
learner-self interactive activities 



Instructional activities for learner-self interaction 


Ranking 

Experiences 


Ranking 

Preferences 


Read over each other’s discussions (both on-going and archived 
discussions) in the online forums 


1 


5 


Apply what have learned in real life situations 


2 


1 


Be given individual problem-solving opportunities such as 
responding to questions independently 


3 


4 


Reflect regularly on what have learned 


4 


2 


Summarize take away key points of major topics or discussions. 


5 


3 


Prepare questions before next class or academic topic 


6 


6 


Write critiques or reflection papers about key course topics or 
concepts 


7 


7 


Keep an electronic study journal 


8 


8 



Online learners prefer receiving prompt feedback, participating in asynchronous lectures and discussions, 
sharing resources and experiences with peers, and participating in team-based task collaborations. It is interesting to 
note that learners prefer to have informal chats with peers more than to engage in class discussions and 
getting/giving feedback to each other although their experiences survey has revealed that informal chats among 
learners only happen sometimes. Another important finding is that learner preferences toward class-level 
asynchronous discussion has ranked much lower even though it is used quite a lot in online learning. In terms of 
learner-self interactions, online learner preferences for reading each other’s discussions ranked much lower although 
it is the most frequently used instructional activity that promotes learner-self interaction. Most of the preferred 
activities are not regularly advocated in online courses. The effectiveness of online courses can improve a lot if 
instructors often ask students to engage in interactive activities. The qualitative data provides a number of detailed 
explanations over some of the “why” questions which are discussed in the following section. 
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Discussions 



Discrepancies exist between learner experiences of and preferences for learner-learner interactions. The 
survey data showed that online students prefer informal chats with peers over engaging in class discussions and 
getting/giving feedback to each other; however, their experience survey revealed that informal chats among learners 
only happened sometimes. This indicates that online learners in this particular study hope to know each other more 
through informal chats. But, in actuality, such a desire to know peers does not have a high priority compared with 
those course-related discussions that students are required to engage in. As a result, informal chat with peers remains 
as a desirable, but not a have-to-do, activity. This point was further illustrated by student interview data. 

. .everybody’s really busy and if I spend time socializing with them, it means that I have that 
much less with my family. . .for the most part. I’m not looking for friendships and social 
opportunities with my peers, except for professional networking and career development.” (Aden) 
“Obviously the course is actually what you are there to do; personal interaction without course 
work is meaningless. It’s nice but it doesn’t really help. . .but there are still people that I keep in 
touch with. . .1 think that is something that I want to do a lot more of, try to keep those 
relationships going.” (William) 

Another finding is that learner preference for class-level asynchronous discussion ranked quite low on the 
list of learner-learner interaction items, even though it was used quite often in online learning. This shows that 
online learners may have had unpleasant experiences with class-level discussions in the past. Since class-level 
discussion is one of the frequently used instructional methods in online education, it is critical to understand why 
students do not prefer to have class-level discussion as much. How can it be improved? These issues were addressed 
in the student interviews. Results indicated that the relatively low preferences for class-level discussion were related 
to large class size, repetitive postings, and the way in which professors set rules and expectations. 

“I don’t like the entire class discussions because so many people go in there and they end up 
posting the same thing that’s already been said 18 times” (Aden) 

“I like [whole class discussions] if the class is small. .. [otherwise] impossible to do it that way. I 
am very frustrated with that. . .Because there are too many people. You cannot have an actual 
conversation with 100 people.” (Jessica) 

“The successful forum discussions that I’ve been involved in, really enjoyed and learned from, the 
professor set ground rules. . .And one of the things that has helped is to understand that the forum 
is not a place to say, yeah that was great. . .you wouldn’t want to read a lot of material that was 
meaningless and probably repetitive of what others have said.” (William) 

Obviously, class-level discussions can become overwhelming if there are too many students in a class. It 
takes extensive time to read through all the discussions in that case. It can be boring and time-wasting if there are 
multiple postings that reflect similar ideas. While asked to provide suggestions for improvement, students 
recommended a number of strategies. 

Dividing up the class into a few groups : 

“One thing that I like ... is combining the individual teams of 3, 4, 5 people into groups of teams 
so there are four or five teams in a group and we just post to that group. So instead of having to 
read 100 people’s posts I just get to see a nice cross section of two or three other teams so it limits 
it to about 20-30.” (Matthew) 

“It is not necessarily the entire class posting. So a class of 70 students maybe there’s a forum for 
35 and there is another forum for 35 and that’s a reasonable number.” (William) 

Professor participation in discussions : 

“I like it as long as professors are involved in discussion. I don’t particularly like it when they just 
start a topic and let the class kind of go at it.” (Richard) 

“We would like to hear the authority of what it should be or what it could be and that kind of thing. 

So it is like a little direction on the forum. And that I would prefer.” (Sandra) 

Setting clearer rules and expectations : 

“There needs to be rules set up ahead of time in terms of what the expectations are of the 
discussion. If the topics are too general, that is when the discussions become somewhat useless.” 

(Richard) 

“Sometimes it is kind of vague and you don’t always know what you are supposed to say. So it is 
good to have kind of a specific target of what you are supposed to be talking about.” (Paulo) 
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From the student follow up interviews, we could tell that the relatively low preference for class- 
level discussions in this study was not because students thought it was unnecessary, but because such 
discussions were not conducted in an effective way. Students admitted the importance of having class 
discussions. 

“I actually love the forum. I think that is where I learn the most.” 

(Sandra) 

“The process of discussion, debate, interaction and all that are as important as the actual outcome 
because I learn a lot from other peoples view points. What they’re saying, that’s very useful.” 

(William) 

Given the critical role of class-level discussions in online learning, instructors should always consider how 
to make it more effective and enjoyable to learners. As Raleigh (2000) pointed out, “planning and implementing the 
online discussion is key to avoiding the common problems experienced with online discussions.” I hope the student 
suggestions for forum improvement in this study provide useful insights for effective planning and implementation 
strategies for online instructors to utilize in promoting leamer-leamer interactions. 

Students prefer certain activities over others in learner-self interactive activities. Survey results suggested 
that online learner preferences for reading others’ discussions ranked low, although it was the most frequently used 
instructional activity among a list of activities that promote learner-self interaction. This result could be related to 
the earlier finding of low student preferences for class-level discussions. If there are too many discussion entries, 
multiple postings of the same ideas, and an overwhelming amount of postings, it becomes boring and time 
consuming to read through discussion forums. 

Utilizing what was learned in real life situations, reflecting on what was learned, summarizing key points of 
topics, and solving problems independently were more preferred by online students compared with reading others’ 
postings. The interview data provided further explanations on why students tended to prefer certain activities over 
others in learner-self interactions. Engaging in these self-interactive activities helps students internalize the 
knowledge, makes the learning outcomes more visible, and also helps students solve work-related issues. 

“I mean, I always look at the way what we are learning applies to my current job and constantly 
reflect on it or take notes or you know, figure out ways to make it my own. . .1 would say the 
majority of what I get out of my MBA and my classes is that internalization” (Richard) 

“Yeah, I like to try to apply such and such an idea perhaps to work and I like it when my employer 
knows that I’m applying those principles since he’s paying for my MBA.” (Aden) 

“Yeah, it’s come in pretty handy. . .1 tried to apply what we were learning to my actual job and 
actually came up with some pretty good ideas from that.” (Amid) 

The relatively large gap between preferences for and experiences of learner-self interactive activities 
indicated most of these self interactive activities were not implemented enough in online courses. The effectiveness 
of online courses can improve greatly if instructors encourage students to engage in self-reflective activities. From a 
constructivist standpoint, learning happens when individuals internalize knowledge. Self-reflective activities can 
make knowledge internalization more effective and long-lasting. 

Implications 

The significance of this study is two-fold. First, from a practical point of view, the results of this study can 
provide empirical grounds for understanding student preferences for instructional activities that promote online 
course interactions. Such an understanding can help practitioners modify their instructional strategies to offer a 
satisfactory course. Second, findings can contribute to the knowledge associated with the use of instructional 
activities as mechanisms for generating online course interaction and thus inevitably enrich the body of literature on 
enhancing online course quality in general. To my knowledge, such empirically-based research study is greatly 
needed in the field of online education. 



Conclusions 

People often choose to learn online because of its flexibility and convenience. While accommodating 
typical online learner needs for flexibility and convenience, ways to enhance learning outcomes becomes a major 
concern of current online learning practices. Meeting learner preferences for learning activities definitely serves as a 
starting point for delivering a satisfactory educational experience for busy working professionals who choose to 
learn at a distance. 
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Abstract 

This study investigated the effects of collaboration mode and group composition during a collaborative 
computer-based program. Each of six intact sections of a computer literacy course was assigned to either a face-to- 
face or a virtual, online collaboration treatment condition. Groups consisted of homogeneous lower-ability, 
homogeneous higher-ability, or heterogeneous mixed-ability pairs. The study examined the effects of collaboration 
mode and group composition on individual posttest performance, group project performance, collaborative 
interaction behavior, and attitudes towards the instruction. The findings suggest that both virtual and face-to-face 
collaboration can be effective in achieving learning goals. However, consideration should be given to the 
collaborative structure of the lesson and the type of task in the design of computer-mediated collaborative 
environments. 



Introduction 

The use of the Internet and telecommunication technologies in education has increased in recent years with 
the proliferation of online learning. In 2002, more than two million college students were estimated to be enrolled in 
some form of web-based education. It is estimated that more than 90 percent of U.S. colleges and universities 
offered online options in 2005 (The Chronicle of Higher Education, 2005). 

This movement towards online learning has prompted educational technologists to debate the most 
appropriate role for new technology (Golas, 2002) and motivated an educational paradigm shift from single- 
classrooms to knowledge-building communities of learners (Chou, 2001; Ravits, 1997). Advocates of computer- 
mediated collaboration (CMC) point to constructivism as the theoretical perspective for explaining its effectiveness 
(Kanuka & Anderson, 1998). Theorists suggest that learning is more effective when students are able to discuss 
their ideas, experiences and perceptions with peers (Jonassen & Kwon, 2001; Kanuka & Anderson, 1998). Some 
researchers have indicated that the flexibility and technological support for such interactions available in computer- 
mediated environments point to collaborative learning strategies as a promising means to implement new technology 
(Laffey, Tupper, Musser, & Wedman, 1998; Pena-Shaff & Nicholls, 2004; Strijbos, Martens, & Jochems, 2004; 
Oliver & Omari, 2001). Furthermore, theorists have indicated that CMC could have greater effects than other 
modes of interaction on learning in problem-solving, case study, and other higher-order learning situations 
(Adelskold, Alklett, Axelsson, & Blomgren, 1999; Johnston, 1996; Jonassen, Prevish, Christy, & Stavulaki, 1999). 

Researchers have shown that collaborative learning strategies in the classroom setting can positively affect 
learning outcomes, social skill development, and self-esteem (Johnson & Johnson, 1996; Slavin, 1990). Research 
also provides support for the use of collaborative learning strategies when students use computer-based instruction 
(CBI) (Cavilier & Klein, 1998; Dalton, Hannafm, & Hooper, 1989; Klein & Doran, 1999; Hooper, 1992; Hooper & 
Hannafm, 1991; Sherman & Klein, 1995). 

Despite the positive findings for the use of collaborative strategies in CBI, there is little empirical evidence 
to indicate if the positive effects of collaborative learning on achievement transfer to environments where 
communication is mediated by computers. Much of the research on online learning has focused on student rates of 
participation and learner interaction (Macdonald, 2003; Pena-Shaff & Nichols, 2004; Sapp & Simon, 2005; Vrasidas 
& Mclsaac, 1999). According to Hara (2002) and Murphy and Collins (as cited in Uribe, Klein, & Sullivan, 2003), 
research on synchronous CMC has been limited to surveys of students, investigations of the recreational use of 
online chat systems, and the use of these systems for instructional purposes has been explored only through case 
studies. Furthermore, among the studies that have examined achievement in the online environment, many consider 
only one collaborative condition rather than make a comparison of online to face-to-face situations (Berge, 1999; 
Brewer, 2004; Davies & Graff, 2005; Gunawardena & Mclsaac, 2004; Hoskins & van Hooff, 2005; Faffey et al, 
1998). Unless comparisons are made, it is difficult to determine the effect of computer-mediated collaboration 
(CMC) in virtual environments. 
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Another variable that may influence outcomes in collaborative learning settings is ability grouping. Ability 
grouping is the assignment of participants to small groups based on academic ability. Several studies have 
addressed ability grouping in face-to-face collaborative environments. However, there appears to be no single best 
way to divide students into learning groups. Some of these studies suggest that heterogeneous ability-groupings 
assist students of all ability levels with the acquisition of knowledge and the cognitive processing of this knowledge 
(Johnson & Johnson, 1996; Slavin, 1993). But other studies suggest that for the optimal development of thinking 
strategies and maintenance of self-esteem, group members should have similar cognitive abilities (Saleh, Lazonder, 
& De Jong, 2005). 

Researchers who have advocated heterogeneous grouping by ability suggest numerous potential benefits 
for the higher-ability partners in these groups (Johnson, Johnson, & Holubec, 1996; Sharan & Sharan, 1992; Slavin, 
1990). Heterogeneous groupings may support the academic achievement of high ability students by providing 
opportunities for deeper cognitive processing through the explanation of their own understanding to their partners. 

It is also suggested that higher-ability students acquire increased motivation and improved self-confidence. 

However, other studies have shown a negative or no impact on higher-ability students when paired in heterogeneous 
dyads (Hooper, 1992; Hooper & Hannafm, 1991; Sherman & Klein, 1995). Other studies suggest that homogeneous 
grouping may be the best when working collaboratively. Swing and Peterson (1982) found that students of average 
ability perform better in homogeneous groups than in heterogeneous groups. Hooper and Hannafm (1988, 1991) 
found that heterogeneous groupings have a negative impact on the achievement of higher-ability students. This is 
consistent with the findings of Sherman and Klein (1995) in which the confidence of higher-ability students was 
negatively affected by heterogeneous grouping. 

Research also suggests that the efficacy of collaborative groupings is related to the interactions students 
exhibit when working in small groups. According to Sherman and Klein (1995), ’’Studies in which group member 
interactions have been recorded and analyzed indicate that achievement and attitude differences are related to the 
type and amount of verbal interactions between students” (p. 6). Webb (1989) reported that students in small groups 
who give or receive explanations during a lesson learn more from the lesson than those who do not. King (1989) 
found that small groups that asked task-related questions, discussed strategy, and elaborated solutions were more 
successful at problem solving than groups that did not exhibit these interaction behaviors. Gunawardena and Zittle 
(1997) noted that social interaction between learners could contribute to learner satisfaction and frequency of 
interaction in online or web-based instruction. 

More recent research has shown that the mode of collaboration also impacts the frequency and quality of 
student interactions in collaborative environments. While some studies have found that participation and interaction 
is reduced in computer-mediated environments when compared to a face-to-face condition (Fahy, Crawford, & 

Aely, 2001; Vrasidas & Mclsaac, 1999), much of the research has indicated that the quality of learner-learner 
interactions in a computer-mediated environment may actually be better than interaction in a face-to-face 
environment (Oliver & Omari, 2001; Sapp & Simon, 2005; Uribe et al, 2003). 

The present study was designed to investigate the effects of two levels of collaboration mode (virtual or 
face-to-face) and the composition of groups (homogeneous higher-ability, homogeneous lower-ability, or 
heterogeneous mixed-ability dyads) within the context of computer-mediated instruction. Data on individual 
performance, group performance, attitudes, and time on task were collected for all participants. Data were also 
collected on the quality and frequency of learner-learner interactions in both the face-to-face and virtual 
collaborative condition. This study addressed the following research questions: 

1 . What is the effect of collaboration mode (virtual or face-to-face) on individual posttest performance, group 
project performance, attitude, and group member interaction in a computer-mediated, collaborative setting? 

2. What is the effect of group composition (homogeneous higher-ability, homogeneous lower-ability or 
heterogeneous mixed-ability dyads) on individual posttest performance, group project performance, 
attitude, and group member interaction in a computer-mediated, collaborative setting? 

3. What is the interaction effect of collaboration mode and group composition on individual posttest 
performance, group project performance, attitude, and group member interaction in a computer-mediated, 
collaborative setting? 



Method 



Participants 

The participants for this study were 120 undergraduate preservice teachers enrolled in a computer literacy 
course at a state university in the northwestern United States. All participants were completing prerequisite 
requirements for entry into the upper division teacher certification program. The sample included students enrolled 
in six sections of the course. Participants were predominantly Caucasian female (69%), with a mean age of 22, from 
all major content areas. 
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The computer literacy course met twice a week for 75 minutes, and introduced basic technology skills in 
word processing, spreadsheets, databases, and presentation software. The Blackboard course management system 
was used as a supplement to face-to-face instruction in the course. Assignments were related to the basic function of 
each application, and face-to-face collaborative groups were often used during activities related to integration of 
technology into the classroom. In addition, this course prepared participants to take a state-mandated technology 
competency exam, which must be passed to receive credit for the course. 

Materials 

A computer-based instructional module on the basic functions of Microsoft Excel and the application of 
spreadsheets in the classroom was written for this study. It consisted of three parts: an introduction, a practice 
problem, and a group application project. 

The program was specifically designed for use by collaborative dyads. Two scores contributed to the grade 
each student earned from the CBI lesson - one score was from a group application project each dyad completed as 
part of the program and the other score was from a posttest each individual completed at the end of instruction. 
Students were informed in the introduction that each score would contribute equally to their final score for the 
spreadsheet assignment. 

The introduction contained 16 screens which provided information for students’ successful use of the 
program. It began with a brief demographic questionnaire, followed by instructions for using the program such as, 
where to click to advance to the next screen, and how to return to previous instruction. The introduction then 
presented a description of the collaborative nature of the lesson, the goals and objectives of the program, and an 
overview of spreadsheets, and encouraged students to help each other learn. A collaborative skills review reminded 
students to: give explanations when their partner asked for help, ask about their partner’s perceptions of the 
concepts, and wait to proceed through the instruction until their partner is ready. In addition, the review emphasized 
the importance of summarizing and listening when working collaboratively. The introduction concluded by 
informing students they would be learning about the basics of Excel, and would be required to develop a spreadsheet 
to solve some problems. 

The practice problem component of the instruction consisted of 70 screens containing a classroom based 
scenario. Students assumed the role of a classroom teacher with the task of developing a class gradebook. The 
gradebook activity required students to determine several elements to include: class title, student names, 
assignments (homework, projects, quizzes, and exams), and the contribution of each assignment to the final grade. 
The gradebook would be formatted to calculate: student performance in each assignment category, student 
performance for the grading period, students’ final grades as a percentage and letter, the class average for each 
assignment, and the highest and lowest grades for each assignment. The gradebook would also include a graph. 
Completed gradebooks were then saved to each student’s network folder and the file path submitted to the 
researcher through the CBI, permitting the group to proceed to the group application project. 

To encourage collaboration throughout the instruction, the practice problem was structured in two parallel 
tracks following a modified Jigsaw procedure (Aronson, Blaney, Sikes, & Snapp, 1978). Each track contained 
different skills required during the practice exercise so that one student could not receive all of the information 
necessary to complete the practice exercise independently. Once a student selected a track, they were unable to 
access the other track. This required each student to learn skills from, and teach skills to, their partner. Tracks were 
accessed by clicking the appropriate button. 

Both tracks presented the student with a list of concepts and terms that should be defined and mastered. The 
tracks contained brief explanations and screen captures to illustrate the definition and function of each element in the 
instruction. Students were informed of the objectives and content within both tracks to ensure that all participants 
were aware of the skills required of the individual posttest. The program also contained a reminder that all of the 
skills included in both content tracks were required of the group application project. 

A coder track, consisting of 32 instructional screens, covered the use of formulas and functions in 
Microsoft Excel. A designer track, consisting of 23 instructional screens, covered the formatting of a spreadsheet 
for efficient use. The remaining 15 screens were common to both tracks. These screens concluded the practice 
activity by providing instruction and practice on the Excel chart wizard. 

The group application project consisted of a second spreadsheet activity, which required the same skills 
covered in the gradebook practice section. The group application project comprised the final seven screens of the 
CBI. No additional instruction was provided for this activity, but students were able to access their previous content 
track for review. Students had to determine on their own how best to complete the project. One application project 
was submitted per group. 



357 




Procedures 

This study included six different treatment groups. Each of the six intact sections of the computer literacy 
course was assigned to either a face-to-face or virtual collaboration treatment condition. Using randomized block 
sampling within each class, participants were blocked by ability and assigned to dyads in one of three ability 
compositions (homogeneous low, homogeneous high or heterogeneous). Ability blocking was based upon 
performance on a general computer ability pretest. A one-way analysis of variance (ANOVA) conducted on the 
pretest scores showed no significant differences between class sections prior to the study, F(5,136) = .08, p> .99. 
Due to the composition of each section, 38 students were assigned to higher-ability dyads, 40 to lower-ability dyads, 
and 42 to mixed-ability dyads. There were a total of 138 students at the beginning of the study. Data from 18 
students were unusable due to absences during the study. 

Dyads were given three, 75-minute class periods to complete the program and assessments. Student 
interactions were collected for each dyad during the first two days of the treatment by two trained research 
assistants. The frequency of these interactions were classified and recorded on an observation form. 

On the first day of the treatment period, students in the face-to-face condition were verbally informed of 
their dyad assignments and asked to sit at a workstation next to their partner. Students were then asked to execute 
the installed CBI. Students in the virtual condition were informed they were participating in a simulation of a virtual 
environment, so all communication would take place using the synchronous chat feature of Blackboard. Partners in 
the virtual condition were not seated in proximity of one another or aware of each others’ identity until logging in to 
Blackboard. Participants in both conditions were instructed to run Microsoft Excel along with the CBI. 

The second day of instruction was nearly identical to the first. Students in both conditions were asked to 
return to the workstation they used the previous class period and resume the lesson. Students in the virtual condition 
were again be reminded they had to login to Blackboard to communicate with their partner. At the conclusion of the 
second class period groups were required to submit their final projects. Any students finishing before the end of the 
period were excused. 

An individual, computer-based, multiple-choice posttest covering the instructional material was 
administered to each student on the third day on instruction. Students also completed a Likert-type attitude survey, 
and five students from each treatment condition were randomly selected to participate in a short interview. 

Data Collection Instruments 

Six measures were used to collect data in the study. Participants completed a general computer ability 
pretest for assignment to ability groups. During instruction, student interactions were collected. Following 
instruction, participant performance was measured by a project scoring rubric and a posttest. Students also 
completed an attitude survey and interview after instruction. 

Several weeks prior to the implementation of the study, a 25-item, computer-based, multiple choice pretest 
measuring general computer ability was administered to the participants. This pretest was used to determine student 
ability for assignment to dyads. The pretest contained five knowledge items from five of the test categories: the 
computing environment, word processing, presentation software, spreadsheets, and databases. Each pretest item 
was worth one point. All pretests were scored by the primary researcher. 

A 25-item, multiple choice, computer-based posttest was administered during the week following the 
spreadsheet CBI. The test included items based on the program objectives, measuring knowledge and skills covered 
in the instructional program. Participants were required to identify various spreadsheet functions, terms, and the 
output of a given formula. Each posttest item was worth one point, and all posttests were scored by the primary 
researcher. The split-half reliability coefficient of the posttest was .88. 

An attitude survey was developed to measure students’ reactions to the instruction. It contained 18, five- 
choice Likert-type items (4 - strongly agree, 0 - strongly disagree) and three open-ended questions. The survey 
included three sections, delivery system, topic, and collaborative work. 

A six-item interview protocol was developed by the researcher to follow the attitude survey. Five 
participants from each treatment condition were asked both forced-response and open-ended questions related to 
their opinion of the program, the helpfulness of features of the program, and the perception of collaborative learning 
in each treatment condition. 

The primary researcher developed a rubric to evaluate participant performance on the group project. Due 
to the nature of the problem, there was not a single correct answer. Therefore, students were evaluated on the 
inclusion of content, the accuracy of their calculations, and the format of their output. For inclusion of content, 
participants were evaluated on their efficiency in using the data provided. For accuracy of calculations, participants 
were evaluated on conducting the correct calculations and if all equations were accurate. For output, participants 
were evaluated on their clarity in formatting the spreadsheet. The group project spreadsheets were blind scored by 
one of two evaluators to prevent bias. Inter-rater reliability for a random sample of 15 student projects was .90. 
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Student interactions were collected for each dyad during the study using two different means. In the face- 
to-face condition, dyads were observed by trained research assistants. Each instance of collaboration was indicated 
on an observation form. Raters identified each dyad by number, but were unaware of the dyad’s ability composition 
to avoid any bias. Each dyad was observed in two minute intervals at various points throughout the program 
comprising a minimum of 20 minutes of observation during the instruction. 

Interactions in the virtual condition were captured using the virtual classroom session log feature of 
Blackboard. The log files for each dyad were exported to a database for analysis by the primary researcher, and 
examined for the same collaborative behaviors as the face-to-face dyads. To ensure an equivalent comparison of 
interactions between the face-to-face and virtual dyads, time stamps for each interaction were used to reconstruct the 
class session on a timeline. Two-minute intervals were then systematically categorized to simulate the observations 
of the face-to-face dyads. 

Design and Data Analysis 

This study used a quasi-experimental, posttest-only control group design. It was a 3 (group composition: 
higher-ability dyads, lower-ability dyads, and heterogeneous dyads) by 2 (collaboration mode: face-to-face and 
virtual) factorial design. 

Group measures were analyzed using analysis of variance (ANOVA) on group project performance. 
Individual measures were analyzed using ANOVA on posttest performance and separate 3x2 MANOVA’s on each 
factor of the attitude survey. ANOVA ’s on each dependent variable were conducted as follow-up tests to the 
MANOVA’s. Follow-up univariate and Tukey HSD analyses were used where appropriate. To control for Type 1 
error, each follow-up ANOVA was tested at the .01 level. Interaction frequency data were analyzed using Chi 
square analyses. Interview responses were categorized and reported by theme. 

Results 



Posttest Performance 

Means and standard deviations for individual posttest performance are reported in Table 1. The mean 
posttest score for all participants was 18.15 (SD = 2.84) out of a possible score of 25. The mean posttest score was 
18.69 (SD = 2.70) for students in the face-to-face collaborative condition and 17.57 (SD = 2.90) for students in the 
virtual collaborative condition. Table 1 also shows that the mean posttest score was 16.70 (SD = 2.30) for students 
in homogeneous lower-ability dyads, 18.40 (SD = 2.80) for students in heterogeneous mixed-ability dyads, and 
19.39 (SD = 2.78) for students in homogeneous higher-ability dyads. 

Table 1 



Means and standard deviations for individual posttest 

Group Composition 



Collaboration Mode 


EL 


HH 


H/L 


Total 


Face-to-face 




Mean 


17.00 


20.25 


19.00 


18.69 




(SD) 


(2.64) 


(2.05) 


(2.34) 


(2.70) 




n 


22 


20 


20 


62 


Virtual 




Mean 


16.33 


18.44 


17.86 


17.57 




(SD) 


(1.82) 


(3.20) 


(3.12) 


(2.90) 




n 


18 


18 


22 


58 


Total 




Mean 


16.70 


19.39 


18.40 


18.15 




(SD) 


(2.30) 


(2.78) 


(2.80) 


(2.84) 




n 


40 


38 


42 


120 



Note: The maximum score possible was 25 points. 
EL = Homogeneous lower-ability dyads 
H/L = Heterogeneous (mixed-ability) dyads 
HH = Homogeneous higher-ability dyads 
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A 3 X 2 analysis of variance (ANOVA) was conducted to examine the effect of group composition and 
collaboration mode on individual posttest performance. ANOVA indicated a significant main effect for group 
composition, F(2,l 14) = 10.79,/? < .001, partial = .16. Tukey HSD pairwise comparisons conducted as a follow- 
up to this main effect revealed that individual posttest scores for participants in homogeneous lower-ability dyads 
were significantly lower (/? < .01) than scores for participants in the homogenous higher-ability dyads and those in 
the heterogeneous mixed-ability dyads. There was not a significant difference between participants in the mixed- 
ability dyads and those in the higher-ability dyads. 

ANOVA also indicated a significant main effect for collaboration mode, F(l,l 14) = 6.43,/? < .01, partial f[ 
= .05. Participants in the face-to-face collaborative condition significantly outperformed those in the virtual 
condition. ANOVA did not reveal a significant interaction between group composition and collaboration mode. 

Group Project Performance 

Means and standard deviations for group project performance are reported for dyads in Table 2. The mean 
project score for all dyads was 21.77 (SD = 1.91) out of a possible score of 24. The mean project score was 21.23 
(SD = 2.01) for dyads in the face-to-face collaborative condition and 22.34 (SD = 1.59) for dyads in the virtual 
collaborative condition. Table 2 also shows that the mean project score was 20.65 (SD = 1.79) for students in 
homogeneous lower-ability dyads, 22.38 (SD = 1.66) for students in heterogeneous mixed-ability dyads, and 22.26 
(SD = 1.85) for students in homogeneous higher-ability dyads. 

Table 2 



Means and standard deviations for group project performance 



Collaboration Mode 


Group Composition 
LL HH H/L 


All Groups 


Face-to-face 




Mean 


19.64 


22.00 


22.20 


21.23 




(SD) 


(1.36) 


(1.93) 


(1.96) 


(2.01) 




n 


11 


10 


10 


31 


Virtual 




Mean 


21.89 


22.56 


22.55 


22.34 




(SD) 


(1.45) 


(1.74) 


(1.44) 


(1.59) 




n 


9 


9 


11 


29 


Total 




Mean 


20.65 


22.26 


22.38 


21.77 




(SD) 


(1.79) 


(1.85) 


(1.66) 


(1.91) 




n 


20 


19 


21 


60 



Note: The maximum possible score was 24 points. 

LL = Homogeneous lower-ability dyads 
H/L = Heterogeneous (mixed-ability) dyads 
HH = Homogeneous higher-ability dyads 

A 3 X 2 analysis of variance (ANOVA) was conducted to evaluate the effect of group composition and 
collaboration mode on group project performance. ANOVA indicated a significant main effect for group 
composition, F(2,54) = 5.85,/? < .01, partial x\ = .18. Tukey HSD pairwise comparisons conducted as a follow-up 
to the main effect revealed that group project scores for dyads in homogeneous lower-ability dyads were 
significantly lower (/? < .01) than scores for participants in the heterogeneous mixed-ability dyads and those in 
homogenous higher-ability dyads. There was no significant difference between the other group compositions. 

ANOVA also indicated a significant main effect for collaboration mode, F(l,54) = 5.93,/? < .05, partial f[ 
= .10. Participants in the virtual collaborative condition significantly outperformed those in the face-to-face 
condition. ANOVA did not indicate a significant interaction between group composition and collaboration mode. 
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Student Attitudes 

Attitude scores were based on a 5 -point, Likert-type scale (4 - strongly agree, 0 - strongly disagree). These 
data indicated that most students felt the computer program was easy to navigate (M = 3.18, SD = .79) and that they 
are better prepared to use spreadsheets after completing the program (M = 3.01, SD = .76). Most students thought 
the spreadsheet is a useful tool to know (M = 3.61, SD = .54), and the spreadsheet skills learned will help in my 
career (M = 3.35, SD = .62). Students also generally liked their assigned partner (M = 3.25, SD = .81). However, 
students did not respond as positively to working with a partner over working alone (M = 1.84, SD = 1.28) and did 
not feel that they learned the material better working with a partner than they would have on their own (M = 2.25, 
SD= 1.28). 

Three separate 3x2 multivariate analyses of variances (MANOVA) were conducted to determine the effect 
of group composition and collaboration mode on each of the attitude factors (delivery system, topic, and 
collaborative work). 

MANOVA conducted on the items related to the delivery system indicated a significant main effect for 
collaboration mode, Wilks’s A= .74, F(7,101) = 5.17,/? <.001. MANOVA did not show a significant main effect 
for group composition or an interaction between group composition and collaboration mode. Follow-up univariate 
analyses indicated a significant difference between collaboration modes on five of the seven items (/? < .01). In all 
cases, students in the face-to-face collaboration mode responded more positively toward the delivery system than 
those in the virtual collaboration mode. 

MANOVA conducted on the items related to the topic indicated a significant main effect for collaboration 
mode, Wilks’s A= .89, F(4,105) = 3.14,/? <.05. MANOVA did not show a significant main effect for group 
composition or an interaction between group composition and collaboration mode. 

Follow-up univariate analyses indicated a significant difference between collaboration modes on two of the 
four items (p<.01). In both cases, students in the face-to-face collaboration mode responded more positively toward 
the topic than those in the virtual collaboration mode. 

MANOVA conducted on the items related to collaborative work indicated a significant main effect for 
collaboration mode, Wilks’s A= .60, F(7,101) = 9.48,/? <.001. MANOVA did not show a significant main effect for 
group composition or an interaction between group composition and collaboration mode. Follow-up univariate 
analyses indicated a significant difference between collaboration modes on five of the seven items. In all cases, 
students in the face-to-face mode responded more positively toward collaborative work than those in the virtual 
mode. 

Approximately 91% of the study participants who completed the Likert portion of the attitude survey also 
responded to three open-ended questions. When asked what they liked best about the program, 24 of 96 respondents 
mentioned that the CBI was interactive. For example, a respondent indicated that they liked the program because, 

“It was interactive; we could move at our own pace. I could go back if I didn’t understand something.” Twenty-two 
participants mentioned the helpful practice, 16 mentioned the step-by-step presentation, and 16 mentioned the access 
to different tracks. When asked what they liked least about the program, 30 respondents indicated the inability to 
copy from the program into Excel, 24 participants mentioned using chat, and 16 responded working with a partner. 

Finally, when asked about how to improve the program, 24 of 96 respondents indicated that the program 
could be improved by not using partners, 20 students wanted more practice in the program, 16 students mentioned 
not using chat, and 12 mentioned increasing the discussion of functions in the program. 

Student Interviews 

Five participants from each treatment condition were interviewed to determine their opinions of the 
program (n = 30). Participants were first asked about their opinion of the program. Fourteen of the 15 students in 
the face-to-face collaborative condition and 12 of 15 students in the virtual condition indicated they liked the 
program. Many responses were similar to the following: 

I felt that the computer program was a good way to learn about spreadsheets. It not only taught the basics, 

but had us apply them at the same time. I liked it because it was informative, creative, and taught the 

basics of spreadsheets. 

Other students said they liked the program because of the step-by-step presentation of information, or because 
students were required to teach what they had learned. 

When asked about which parts of the program were the most helpful, 9 of 1 5 students in the face-to-face 
condition and 8 of 1 5 students in the virtual condition indicated the individual instruction component was the most 
helpful. Six students in the virtual condition said the practice project was the most helpful, while five students in the 
face-to-face condition mentioned the group project. When asked about which parts of the program were least 
helpful, seven students in the face-to-face condition and four students in the virtual condition indicated the inability 
to copy from the program into Excel was the least helpful component. Furthermore, four students in the virtual 
condition identified the Internet chat requirement as the least helpful. 
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Responses to questions about working with a partner revealed that 10 of 15 students in both the face-to-face 
collaborative condition and the virtual collaborative condition liked working with a partner. One of the students in 
the face-to-face condition made the following representative statement: 

I did prefer working with a partner; it was helpful to make sure I was doing things correctly, and for getting 
new ideas on our second project. I think the collaboration made things better for both of us because we 
were able to bounce ideas off each other. 

A student in the virtual condition said, “It was fim to work with a partner in the spreadsheet lesson. I think it helped 
me better understand what I needed to know by having to explain it to someone else.” Conversely, 5 of 15 students 
in both the face-to-face and the virtual collaborative conditions indicated they did not like working with a partner. 
Students responding less positively towards working with a partner found it difficult to, “keep going back to explain 
what I just learned”, and indicated that working with a partner, “left a few gaps in my knowledge.” 

When asked about how the lesson was used to learn about spreadsheets, 17 of 30 respondents said they 
simply followed the instructions. Five of the 15 students in the virtual collaborative condition mentioned that they 
relied on learning from their partner, while only 1 of 1 5 students in the face-to-face condition mentioned learning 
from their partner. 

Finally, students were asked about their opinion of using a similar program in the future. Twelve of 15 
students in the face-to-face condition and 11 of 1 5 students in the virtual condition indicated they thought it would 
be a good idea to use a similar lesson in the future. Three students in each condition thought that it should depend 
on the subject. For example, one student said, “I don't think it would be useful for every application. It is really 
only good for basic procedural information.” Only one student indicated that they thought it would be a bad idea to 
use a similar program in the future. 

Student Interactions 

Interaction behaviors were grouped into the five categories of questioning, answering, encouraging, 
discussing, and off-task. Separate chi-square analyses were conducted to determine the effect of group composition 
and collaboration modes on the number of interactions for each category. No significant differences were found 
within any of the interaction categories between ability groups. However, chi-square analyses indicated that 
students in the virtual collaborative condition asked more questions of their partners, = (1, N = 29) = 5.13,/? = 
<.05, and exhibited more off-task interactions = (1, N = 29) = 9.92,/? = <.01, than those in the face-to-face 
collaborative condition. No other significant differences were found for interaction behaviors. The observed 
instances of student interactions that occurred during the instructional program are reported in Table 3. 

Table 3 



Instances of student interactions 

Group Composition Mode 



Interactions 


HH 


LL 


H/L 


F2F 


V 


Questioning* 


120 


145 


I6I 


188 


238 


Answering 


90 


98 


130 


145 


173 


Encouraging 


51 


39 


58 


63 


85 


Discussing 


185 


132 


175 


258 


234 


Off-task* 


36 


61 


71 


64 


104 


Total Interactions 


482 


475 


595 


718 


834 



Note: Total number of each interaction behavior for 60 dyads 
in 20 minutes of elapsed time observed in two-minute intervals. 
LL = Homogeneous lower-ability dyads 
H/L = Heterogeneous (mixed-ability) dyads 
HH = Homogeneous higher-ability dyads 
*p <.05 for mode 



Discussion 

This study examined the effects of collaboration mode and group composition on individual posttest 
performance, group project performance, student attitudes, and collaborative interaction behaviors. Students 
completed a collaborative computer-based program while communicating either face-to-face or by using 
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synchronous chat in a virtual, online setting. All participants were assigned to one of three group compositions 
based on general computer ability. These dyads consisted of homogeneous lower-ability, homogeneous higher- 
ability, or heterogeneous mixed-ability pairs. 

Performance 

Results indicated that participants in the face-to-face collaborative condition performed significantly better 
on the individual posttest than those in the virtual online condition. This finding likely occurred because face-to-face 
students found it easier to share information throughout the lesson than virtual students. This explanation is partially 
supported by results from the attitude survey which revealed significant differences in favor of students in the face- 
to-face condition for the following items: (1)1 was able to adequately communicate with my partner; (2) My partner 
taught me what I needed to learn; (3) The checkpoints in the program helped my partner and me communicate; and 
(4) I liked the system for communicating with my partner. 

Furthermore, observations of student interactions revealed that several face-to-face dyads used visual cues 
such as pointing to their screen to provide an explanation to their partner or to acknowledge understanding. Hara 
(2002) indicated that students are required to make assumptions about meaning when they collaborate in virtual 
environments because of a lack of visual cues obvious in face-to-face communication. Others suggest that student 
misconceptions may occur due to a lack of nonverbal interventions to signal misunderstandings and that students 
can become disoriented without visual anchors in a virtual environment (Ruberg, Moore, & Taylor, 1996; Sapp & 
Simon, 2005). 

Observations in the current study also revealed that in a few instances, students in face-to-face dyads traded 
seats with their partner so that both students could work individually through each track, avoiding collaboration 
altogether. Slavin (1995) suggested that individual learning strategies are better than cooperative strategies when 
students are required to learn facts and procedures. The multiple-choice posttest in the current study required 
individual students to identify terms, spreadsheet functions, and the output of a given formula. 

In contrast to findings for the individual posttest, dyads that collaborated virtually performed significantly 
better on the group project than those who collaborated face-to-face. The collaborative interactions of students in the 
virtual dyads likely influenced their scores on the group project. Observations conducted during the study revealed 
that dyads in the virtual condition exhibited significantly more questioning behaviors than dyads in the face-to-face 
condition. As the lesson progressed, the frequency of interaction increased for virtual dyads while it decreased for 
face-to-face dyads. Several students in the face-to-face condition were observed working independently on the group 
project. 

Other researchers have found that student interactions influence learning and performance in collaborative 
settings. Hooper & Hannafm (1991) demonstrated that questioning contributes to learning in collaborative groups. 
King (1989) found small groups that asked task-related questions were more successful at problem solving than 
groups that did not exhibit such interaction behaviors. In a comparison of computer-mediated groups, Sherman and 
Klein (1995) reported that dyads exhibiting more helping behaviors such as asking and answering questions 
performed better than dyads exhibiting significantly fewer helping behaviors. 

The interaction requirements for an inquiry type project, such as the group project in this study, may be 
better met by virtual collaboration than face-to-face. Theorists have discussed advantages of virtual over face-to-face 
collaboration for group problem-solving tasks. Jonassen et al. (1999) asserted that CMC environments are better 
suited for problem-solving activities. The process of writing and reflecting may encourage higher level learning 
such as analysis, synthesis, and evaluation, and promote clearer and more precise communication (Garrison, 1997; 
Jonassen & Kwon, 2001). A study by Uribe et al. (2003) found that computer-mediated groups experienced 
performance benefits from the medium when performing an ill-structured problem solving task. 

In addition to the findings for collaboration mode, group composition had a significant impact on 
individual posttest scores and group project performance in the current study. As expected, students assigned to 
higher-ability dyads performed significantly better on both performance measures than students assigned to lower- 
ability dyads. Furthermore, mixed-ability dyads performed significantly better on both performance measures than 
students assigned to lower-ability dyads. 

Student Attitudes 

Turning to attitudes, participants were generally positive about the delivery system and topic of the 
computer-based program used in this study. However, attitudes toward collaboration were less positive. Most of the 
Likert-type survey items asking about collaborative work were rated lower than items about the delivery system and 
the topic. Furthermore, open-ended survey items revealed that many students disliked working with a partner and 
thought the program could be improved by eliminating collaboration. 

These results likely occurred because the collaborative structure used in the study placed too many 
limitations on students’ ability to interact naturally, especially for those in the virtual collaborative condition. 
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Results indicated that students in the virtual condition were significantly less positive than students in the face-to- 
face condition toward the delivery system, the topic, and collaborative work, with significant differences occurring 
on 12 of the 18 Likert-type items. Open-ended items revealed that half of the respondents in the virtual condition 
identified communicating via synchronous chat as the thing they liked least about the program. In addition, almost 
one-third of the virtual students who participated in follow-up interviews cited the chat system as the least helpful 
part of the program. 

These findings are consistent with results from other studies. Uribe, et al. (2003) found that participants did 
not like virtual collaboration due to the difficulties of communicating via computer. Others have reported that 
students in virtual groups were less satisfied than those in face-to-face groups with instruction received from their 
partner (Olaniran, Savage & Sorenson, 1996: Warkentin, Sayeed, & Hightower, 1997). 

Student Interactions 

As discussed above, virtual dyads were observed to have significantly more questioning behaviors than 
face-to-face dyads. In addition, virtual dyads exhibited significantly more off-task interactions than face-to-face 
dyads. Many studies have detailed the importance of social interaction in computer-mediated communication 
(Chen, 2005; Jung, Choi, Lim, & Leem, 2002; Savenye, 2005). Anderson and Harris (1997) identified that socially 
oriented factors contribute to the prediction of performance in computer-mediated settings. 

It is likely that the increased off-task behaviors for virtual dyads are a result of the necessity to establish a 
virtual social presence. It is interesting to note that while group composition did not have a significant effect on 
interaction behaviors, mixed ability groups exhibited the highest number of off-task interactions Perhaps 
heterogeneous groups have a greater need than homogeneous groups to establish a social presence. 

Implications and Future Research 

The results of this study have implications for the design and delivery of computer-mediated instruction in 
collaborative environments. Findings suggest that both face-to-face and virtual collaboration can be effective in 
achieving learning goals. However, consideration should be given to the type of learning task and the collaborative 
structure of the lesson when designing computer-mediated instruction. Face-to-face collaboration may be better 
suited than virtual collaboration to environments where the acquisition of well defined facts and procedures is 
desired. Furthermore, virtual collaboration may be better suited than face-to-face collaboration when solving ill- 
structured problems is the desired outcome. 

It should be noted that results were obtained in an environment constrained by a rather rigid collaborative 
structure. Yet to be resolved is the question of what kind of collaborative structuring should be used to support 
positive outcomes in computer-mediated environments. Theorists have argued that ill-structured tasks are best 
addressed in open-ended environments and that well defined tasks are better addressed in more rigid environments 
(Jonassen, et al., 1999; Jonassen & Kwon, 2001). However, the results of this study seem to indicate that task type 
and structure are mediated by the mode of collaboration. 

The current study also suggests that group composition should be considered when forming collaborative 
dyads. Regardless of the mode of collaboration, pairing two lower-ability students has a negative impact on learning 
facts and procedures and on solving problems. Results also confirm findings of other researchers who suggest 
lower-ability students may benefit from being paired with higher-ability students (Saleh, et al.,2005; Slavin, 1993; 
Uribe et al., 2003). 

The interdependence of design considerations such as collaborative structure, task and collaboration mode 
should be further explored. Additional research is needed to determine whether a particular collaborative structure 
is better suited for certain types of tasks in a CMC environment, or if certain tasks are inappropriate for computer- 
mediated collaboration. As stated by Salomon (1999), “The fact that something is technologically possible does not 
imply that it is also educationally desirable” (p. 36). Research should identify the most effective instructional 
practices to promote the learning of various skills. Future research could also address the method for forming 
groups in online and face-to-face collaborative settings. Factors such as group size should be explored to determine 
an optimal size for learning in computer-mediated collaborative environments. 

As the demand for online and distance education expands, more students will be required to work 
collaboratively to learn from computer-mediated instruction. The production of increasingly complex tools for 
virtual collaboration will challenge practitioners to implement the most effective strategies for learning. Educational 
technology researchers should continue to examine the factors that impact learning when students use web-based 
and computer-mediated instruction in collaborative environments. 



364 




References 

Adelskold, G., Alklett, K., Axelsson, R., & Blomgren, G. (1999). Problem-based distance learning of energy issues 
via computer network. Distance Education, 20{\), 129-143. 

Anderson, S. E., & Harris, J. B. (1997). Factors associated with amount of use and benefits obtained by users of a 
statewide educational telecomputing network. Educational Technology Research and Development, 45(1), 
19-50. 

Aronson, E., Blaney, N., Sikes, J., & Snapp, M. (1978). The Jigsaw classroom. Beverly Hills: Sage. 

Berge, Z. L. (1999). Interaction in post-secondary web-based learning. Educational Technology, 59(1), 5-lE 

Brewer, S. A. (2004). Small group learning in an online asynchronous environment. Unpublished doctoral 
dissertation, Arizona State University. 

Cavalier, J. C., & Klein, J. D. (1998). Effects of cooperative versus individual learning and orienting activities 
during computer-based instruction. Educational Technology Research and Development, 55(3), 52-72. 

Chen, H. (2005). The effect of type of threading and level of self-efficacy on achievement and attitudes in online 
course discussion. Unpublished doctoral dissertation, Arizona State University. 

Chou, C. C. (2001). Student interaction in a collaborative distance-learning environment: A model of learner- 

centered computer-mediated interaction (Doctoral dissertation. University of Hawai'i, 2001). Dissertation 
Abstracts International, 3005200 . 

Dalton, D. W., Hannafm, M. J., & Hooper, S. (1989). The effects of individual versus cooperative computer-assisted 
instruction on student performance and attitudes. Educational Technology Research and Development, 

57(2), 15-24. 

Davies, J., & Graff, M. (2005). Performance in e-leaming: online participation and student grades. British Journal of 
Educational Technology, 5d(4), 657-663. 

Fahy, P. J., Crawford, G., & Aely, M. (2001). Patterns of interaction in a computer conference transcript. 

International Review of Research in Open and Distance Learning. Retrieved June 21, 2005, from 
http ://www.irrodl.org/content/v2 . 1 /fahy .html. 

Garrison, D. R. (1997). Self-directed learning: Toward a comprehensive model. Adult Education Quarterly, 48{\), 
18-34. 

Golas, K. (2002). Guidelines for designing online learning. Retrieved June 6, 2005, from Southwest Research 
Institute Web Site: http://www.tss.swri.org/tsd/publications/20021TSEC_ONLINELEARNING.html 

Gunawardena, C. N., & Mclsaac, M. S. (2004). Distance education. In D. Jonassen (Ed.), The handbook of research 
on education communications and technology (2nd ed., pp. 355-395). Mahwah, NJ: Lawrence Erlbaum 
Associates. 

Gunawardena, C. N., & Zittle, F. J. (1997). Social presence as a predictor of satisfaction within a computer-mediated 
conferencing environment. American Journal of Distance Education, 77(3), 8-26. 

Hara, N. (2002). Analysis of computer-mediated communication: Using formal concept analysis as a visualizing 
methodology. Journal of Educational Computing Research, 2d(l), 25-49. 

Hooper, S. (1992). Cooperative learning and computer-based instruction. Educational Technology Research and 
Development, 49(3), 21-38. 

Hooper, S., & Hannafm, M. J. (1988). Cooperative CBI: The effects of heterogeneous vs. homogeneous grouping on 
the learning of progressively complex concepts. Journal of Educational Computing Research, 4, 413-424. 

Hooper, S., & Hannafm, M. J. (1991). The effects of group composition on achievement, interaction, and learning 
efficiency during computer-based cooperative instruction. Educational Technology Research and 
Development, 59(3), 27-40. 

Hoskins, S. L., & Van Hoof, J. C. (2005). Motivation and ability: which students use online learning and what 

influence does it have on their achievement. British Journal of Educational Technology, 5d(2), 177-192. 

Johnson, D. W., & Johnson, R. T. (1996). Cooperative learning and traditional American values: an appreciation. 
NASSP Bulletin, 80(519), 63-66. 

Johnson, D. W., Johnson, R. T., & Holubec, E. (1996). Advanced cooperative learning. Edina, MN: Interaction 
Book Company. 

Johnston, V. L. (1996). Toward a global classroom using computer-mediated communications at UAA. University of 
Alaska Anchorage, Vocational Teacher Report, ERIC Document database (ED3 5 675 9). 

Jonassen, D. H., & Kwon, H. I. (2001). Communication patterns in computer-mediated and face-to-face group 
^xoh\Qm-^oWmg. Educational Technology Research and Development, 49, 35-51. 

Jonassen, D., Prevish, T., Christy, D., & Stavulaki, E. (1999). Learning to solve problems on the Web: aggregate 
planning in a business management course. Distance Education, 29(1), 49-63. 

Jung, L, Choi, S., Lim, C., & Leem, J. (2002). Effects of different types of interaction on learning achievement, 
satisfaction and participation in web-based instruction. Innovations in Education and Teaching 
International, 59(2), 153-162. 



365 




Kanuka, H., & Anderson, T. (1998). On-line interchange, discord, and knowledge construction. Journal of Distance 
Education, 75(1), 51-1 A. 

King, A. (1989). Verbal interaction and problem-solving within computer-assisted cooperative learning groups. 
Journal of Educational Computing, 5(1), 1-15. 

Klein, J. D., & Doran, M. S. (1999). Implementing individual and small group learning structures with a computer 
simulation. Educational Technology Research and Development, 47{\), 97-110. 

Laffey, J., Tupper, T., Musser, D., & Wedman, J. (1998). A computer-mediated support system for project-based 
learning. Educational Technology Research and Development, 46{\), 73-86. 

Macdonald, J. (2003). Assessing online collaborative learning: process and product. Computers & Education, 40, 
377-391. 

Olaniran, B., Savage, G. T., & Sorenson, R. L. (1996). Experimental and experiential approaches to teaching face- 
to-face and computer-mediated group discussion. Communication Education, 45(3), 244-259. 

Oliver, R., & Omari, A. (2001). Student responses to collaborating and learning in a web-based environment. 
Journal of Computer Assisted Learning, 17, 34-47. 

Pena-Schaff, J. B., & Nicholls, C. (2004). Analyzing student interactions and meaning construction in computer 
bulletin board discussions. Computers & Education, 42, 243-265. 

Ravits, J. (1997). An ISD model for building online communities: Furthering the dialog. The Annual Proceedings : 
Selected Research and Development Presentations at the 1997 National Convention of the Association for 
Educational Communications and Technology, 297-307. 

Ruberg, L. F., Moore, D. M., & Taylor, C. D. (1996). Student participation and interaction in the cmc interchange. 
Journal of Educational Computing Research, 75(3), 243-268. 

Saleh, M., Fazonder, A. W., & De Jong, T. (2005). Effects of within-class ability grouping on social interaction, 
achievement, and motivation. Instructional Science, 33, 105-119. 

Salomon, G. (1999). Computer mediated conferencing in large scale management education. Open Learning, 74(2), 
34-43. 

Sapp, D. A., & Simon, J. (2005). Comparing grades in online and face-to-face writing courses: Interpersonal 
accountability and institutional commitment. Computers and Composition, 22, 471-489. 

Savenye, W. C. (2005). Feaming technology integration from video cases: Development and research issues. In M. 
Orey, J. McClendon, & R. M. Branch (Eds.), Educational media and technology yearbook (30 ed., pp. 
110-126). Westport, CN: Fibraries Unlimited. 

Sharan, Y., & Sharan, S. (1992). Expanding cooperative learning through group investigation. New York: Teachers 
College Press. 

Sherman, G. P., & Klein, J. D. (1995). The effects of cued interaction and ability grouping during cooperative 
computer-based science instruction. Educational Technology Research and Development, 45(4), 5-24. 

Slavin, R. (1993). Ability grouping in the middle grades: Achievement effects and alternatives. Elementary School 
Journal, 95(5), 535-552. 

Slavin, R. E. (1990). Point-counterpoint: ability grouping; cooperative learning and the gifted. Journal for the 
Education of the Gifted, 14(3), 3-8. 

Slavin, R. E. (1995). Cooperative learning: theory, research, and practice (2nd ed.). Needham Heights, MA: Allyn 
and Bacon. 

Strijbos, J. W., Martens, R. E., & Jochems, W. M. (2004). Designing for interaction: six steps to designing 
computer-supported group-based learning. Computers & Education, 42, 403-424. 

Swing, S. R., & Peterson, P. L. (1982). The relationship of student ability and small group ability and small group 
interaction to student achievement. American Educational Research Journal, 79(2), 259-274. 

The Chronicle of Higher Education. (2005). IT by the numbers. Retrieved June 6, 2005, from The Chronicle of 
Higher Education Web Site: http://chronicle.eom/weekly/v50/i21/21b02901.htm 

Uribe, D., Klein, J. D., & Sullivan, H. (2003). The effect of computer-mediated collaborative learning on solving ill- 
defined problems. Educational Technology Research and Development, 57(1), 5-19. 

Vrasidas, C., & Mclsaac, M. S. (1999). Factors influencing interaction in an online course. The American Journal of 
Distance Education, 75(3), 22-36. 

Warkentin, M., Sayeed, L., & Hightower, R. T. (1997). Virtual teams versus face-to-face teams: An exploratory 
study of a web-based conference system. Decision Sciences, 25(4), 975-996. 

Webb, N. M. (1989). Peer interaction and learning in small groups. International Journal of Educational Research, 
13, 21-40. 



366 




Designing ePortfolios for Learning Outcomes Assessment: Course to 
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Introduction 

Accountability demands to provide evidence of student learning has made universities focus on outcomes 
assessment. This emphasis on assessment has brought to the forefront the use of electronic portfolios to 
showcase what students have learned as a result of their university studies over time. Universities are being 
called to document new learning — what students know now that they did not know upon entering the 
university. Due to this outcomes emphasis alternative assessments to the traditional examinations are being 
employed for accountability purposes (Ruhland & Brewer, 2001). Because of the increased interest in 
learning outcomes assessment a divergent paradigm has emerged — assessment 6>/ learning and assessment 
for learning (Barrett & Carney, 2005). Universities find it necessary to use an outcomes assessment process 
and engage students in active, learner-centered activities, problem solving, writing, researching, analyzing, 
implementing, for a student to be a responsible, reflective learner. This process requires timely feedback to 
be provided to the learner to guide the student learning process — or assessment for learning. Assessment of 
learning, the more traditional process, is a summative assessment approach which may only tell part of the 
story of student learning. Astin (1985, 1996) states that students learn when they are in educational 
programs that provide learning environments where students are actively engaged, have high expectations 
and are given ongoing feedback. Current research recognizes the value of students situated in the 
construction of learning. Students involved in the process of learning make a decisive difference in 
promoting positive learning outcomes (Davis & Murrell, 1994). Universities have begun to devise and 
incorporate an assessment process which allows students to convey a better understanding of the depth and 
breadth, a richer story, a holistic picture, of what they know and are capable of doing (Niguidula, 1993). 

The American Association for Higher Education (AAHE) outlined nine principles of good practice for 
assessment of student learning (Astin, Banta, Cross, El-Khawas, Ewell, Hutchings, Marchese, McClenney, 
Mentkowski, Miller, Moran & Wright, 1996). These principles of assessment suggest true formative 
assessment to track students’ process and progress and monitor changes in their attitudes, values and 
thinking over time and involve students in the learning, teaching and assessment process. The principles 
help to inform program quality and allow students to not only demonstrate what they know, but what they 
are able to do with the knowledge. These principles embody the concept of an electronic portfolio. Through 
the use of electronic portfolios, the responsibility of learning is transferred to the students. It allows them to 
be involved and engaged in the learning process and therefore keeps the focus on the learner-centered 
environment. Hewett’s (2005) research states that “as a model for learner-centered classrooms, e-portfolios 
give students ownership and responsibility for their own learning” (p. 27). The EDUCAUSE Learning 
Initiative definition of electronic portfolios was used in order to have a collective way of thinking 
(University of British as cited in Lorenzo & Ittelson, 2005). This definition describes ePortfolios as 
“personalized, Web-based collections of work, responses to work, and reflections that are used to 
demonstrate key skills and accomplishments for a variety of contexts and time periods” (p. 3). 

Research on student perceptions of electronic portfolios provide evidence that students believe this 
environment enables them to demonstrate competence, critical thinking and personal development through 
reflections and professional growth of knowledge and abilities. Utilizing an electronic portfolio enables 
students to exhibit a constructivist approach to learning through active learner-centered activities such as 
problem solving, writing, researching, analyzing, and implementation of programs. Students are able to 
integrate and make connections from course to course, link items, and see associations as a result of 
reflections (MacDonald, Liu, Lowell, Tsai, & Lohr, 2004). Wilson, Wright & Stallworth (2003) found that 
students prefer to use electronic portfolios to self-evaluate their conceptual knowledge and show their 
ability to connect learning. Students are more engaged as a result of assessment through portfolios and their 
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personal theories, beliefs and practices come together in a cohesive bond. Students recognize that 
portfolios provided a process to showcase their learning and their students’ learning and as a result, 
learning becomes their responsibility. They also note that organization, structure and technological skills 
are developed within a relevant context. 

Teacher education students using an electronic portfolio for reflection say they more clearly understand 
pedagogical knowledge and are effective in the classroom and teaching areas where improvement is needed 
and can be developed. They have an increased self-efficacy and see teaching from a more holistic view- 
relating parts to whole. They believe it is an authentic assessment that chronicles student growth and 
competence. They provide a relevant personal journey and diverse types of evidence which is needed when 
measuring the depth and breadth of performance (Shaver & Avolos, 2004; Skawinski & Thibodeau, 2002). 
Students like the alternative approach to assessment that eportfolios offer, believe it has an impact on their 
learning, and information learned is retained better and longer because of the extensive reflecting on 
principles and application of concepts that is expected in the portfolio development process. They see that 
electronic portfolios provide an instrument for professional growth and acknowledge its usefullness where 
instruction and evaluation can be integrated. Evidence throughout the literature indicates students’ 
perceptions are positive and favorable; however, the one drawback they offer is the time it takes to go over 
the required readings and time spent in developing their portfolio (Struyven, Dochy, & Janssens, 2002). 



Purpose 

The purpose of this paper is to present strategies/methods utilized to design and develop effective practices 
of assessment for and q/* learning using electronic portfolios and the lessons learned upon implementation. 
In addition, the effectiveness of this process after three semesters of implementation based upon program 
learning outcomes and graduate student perceptions of electronic portfolios as a method of assessment will 
be discussed. 



Methods 

Electronic portfolios have come to the forefront as an assessment process that “can enhance teaching, 
learning and assessment practices” (Lorenzo & Ittelson, 2005, p. 3). A regional university in North East 
Texas, in partnership with a local independent school district, used Astin’s framework and the nine 
principles of good practice for assessment as a guide when planning a new and unique cohort master’s 
degree program in Secondary Education. The goal was to provide the best system for documenting student 
learning outcomes and ways to assess the overall program quality. It was important to be able to use an 
outcomes assessment process that would actively engage students where they would be responsible, 
reflective learners and provide assessment information for feedback to guide the student learning process 
(assessment for learning) and inform program goals, objectives and field-based learning experiences for 
quality enhancement (assessment q/* learning). 

Participants in this study were 26 members of a graduate cohort group working on their master’s degree in 
Secondary Education. Feedback from students regarding the effectiveness of the electronic portfolio 
development process was obtained from a survey administered at the end of each semester consisting of 
thirteen Likert scale response questions, and four open-ended questions; two questions addressing the 
benefits and challenges of electronic portfolios and two questions addressing students’ perceptions of 
electronic portfolios as a method of assessment or tool. Descriptive statistics were used to analyze the data 
from student survey responses in order to make comparasions over three semesters of changes in the three 
learning outcomes and responses from the open-ended questions were analyzed via the constant- 
comparative method (Glaser & Strauss, 1967). 
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Discussion 



Strategies to Design, Develop and Implement Electronic Portfolios 

The design, development and implementation of electronic portfolios within the graduate cohort centered 
around the concepts of nine principles of good practice (Astin, et al, 1996) and Astin’s framework (1985, 
1996) which support the notion of formative assessment, continuous feedback and revision, providing 
students with an environment to display evidence of learning via selected and prescribed artifacts, and 
continuous joumaling/reflecting. The design of the portfolio began with the identification of program goals 
to determine what a student needs to know upon completion of the Secondary Education Master’s degree. 
This overall goal identification assisted in determining goals/leaming outcomes for each course within the 
program and the eventual determination of what students need to do in order to demonstrate achievement of 
learning outcomes in each course. Only two to three learning outcomes for each course were identified and 
assessment rubrics were developed and provided to students. 

Within the first semester of implementation, it was quickly evident that students’ technological skills varied 
from novice to expert and as a result, those with a low degree of technological self-efficacy became 
frustrated with the process of developing their portfolios. Instructors believed students would quickly 
develop those technological skills as a result of working within the electronic portfolio and progress has 
been made. Additionally, as a result of the range of technological abilities of students within the cohort, the 
design and creativity of the portfolios differed and it was from these differences that a new issue emerged. 
Instructors realized that an advantage of a portfolio assessment is that students can be evaluated 
individually based upon academic growth and progress from where their portfolio design began to how it 
evolved each semester. Individual progress and comparisons are more evident within the electronic 
portfolio environment. Portfolios chronicle student growth and competence over time through their 
reflection and self assessment allowing a window into their educational journey. 

Sharing portfolios electronically allowed students to view their peers’ work which was a source of 
reassurance, encouragement and inspiration through the sharing of ideas and artifacts. The electronic 
portfolio environment combined with the rubrics developed for each learning outcome also led to increased 
communications among the course instructors and students with opportunities for continuous feedback and 
revision outside of the classroom. 

Initially, the concept of utilizing electronic portfolios as an assessment tool was foreign to both instructors 
and students; however, the implementation process started small and evolved over each semester adding 
new features of the system and new assignments/activities to encourage linking concepts from course to 
course. It is from these activities that the program is striving to provide a holistic and unique image of each 
student documenting their learning process and progress and over the course of three semesters both 
students and instructors have made steady improvements in technological skills and abilities resulting in 
more dynamic portfolios. Acceptance of the utilization of portfolios to assess student learning has also 
continued to grow. These improvements are evident within data collected in relation to the portfolio 
development process and are discussed next. 

Effectiveness of Electronic Portfolio Development Process 

The cohort is starting its fourth course out of six within the master’s program, and to date data on three 
program learning outcomes from three semesters has been collected to determine whether the electronic 
portfolio development process assisted in the development of self-knowledge, technology and 
organizational skills development, and knowledge and skills transfer. The electronic portfolio development 
process is defined as “the design, development and sharing of one’s electronic portfolio which contains 
evidence of ideas, knowledge and skills development as a result of class activities, reflections, 
communications and assignments to utlimately provide a holistic view that demonstrates an individual’s 
growth as an educator” (Chambers & Wickersham, in press). 

The program learning outcome of self-knowledge was defined as an increase in understanding of the 
relationship between theory taught in the graduate program as it pertains to the what, why and how of 
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individual careers. Within this learning outcome, students were asked to compare their growth over time 
through the analysis of their role as an educator each semester. Students also identified the changes that 
needed to be made for their professional growth. Specific areas surveyed within the self-knowledge 
outcome included the ability to view students, peers and supervisors through a different lens, and self 
awareness through journaling reflection exercises within their electronic portfolios as a result of new 
knowledge. 

The program learning outcome of technology and organizational skills development was defined as an 
increase in technological skills and abilities, technological self-efficacy, communication, and organizational 
skills. Students designed and developed their electronic portfolio to provide an individualized perspective 
of their learning process and progress. It was through the process of building their portfolio, that students 
were expected to develop their technological and organizational skills and enhance their communication 
skills through journaling and conveying ideas and understanding of course content. 

The program learning outcome of knowledge and skills transfer was defined as the ability to transfer the 
knowledge and skills developed throughout the master’s program to their classrooms. Specific areas 
surveyed within this learning outcome were students’ ability to identify and transfer specific skills relevant 
to their context and integrate those skills into their teaching based upon new information presented in the 
course. Students were also asked to document examples of changes in their own assessment practices and 
in their students’ learning within the portfolio. 

The focus of this paper is not specifically aimed toward a detailed discussion of these results and will be 
expanded upon in another article; however, it is important to note that after three semesters of 
implementation of the electronic portfolio development process there has been an overall continual 
improvement for items within the three learning outcomes. The following tables provide the percentage 
distributions and comparisons for items within the self-knowledge learning outcome (Table 1), the 
technology and organizational skills development learning outcome (Table 2), and the knowledge and skills 
transfer learning outcome (Table 3) for each semester (SI, S2, S3). 



Table 1 



Self-Knowledge Learning Outcome 







Agree 






Neutral 






Disagree 




Item 


SI 


S2 


S3 


SI 


S2 


S3 


SI 


S2 


S3 


I . Increase Knowledge in 
Teaching 


33% 


58% 


65% 


21% 


19% 


17% 


46% 


23% 


17% 


2. Reflections 


46% 


54% 


57% 


25% 


35% 


35% 


30% 


11% 


8% 


3. Why 


42% 


38% 


39% 


25% 


46% 


49% 


33% 


15% 


13% 


4. What 


46% 


62% 


52% 


21% 


23% 


30% 


33% 


15% 


17% 


*How 


46% 


N/D 


N/D 


21% 


N/D 


N/D 


33% 


N/D 


N/D 


5. Positive impact 




58% 


57% 




27% 


30% 




15% 


13% 


6. View Peers 




65% 


78% 




19% 


22% 




15% 


4% 


7. View Students 




42% 


43% 




35% 


43% 




23% 


13% 


8. View Supervisors 




42% 


57% 




42% 


39% 




15% 


4% 



*Question reframed 
N/D=No data 
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Table 2 



Technology and Organizational Skills Development Learning Outcome 







Agree 






Neutral 






Disagree 




Item 


SI 


S2 


S3 


SI 


S2 


S3 


SI 


S2 


S3 


I. Technology Skills 


58% 


73% 


83% 


30% 


15% 


13% 


12% 


12% 


4% 


2. Confidence 


50% 


65% 


78% 


46% 


19% 


17% 


4% 


15% 


9% 


3. Organization 


33% 


42% 


43% 


38% 


35% 


52% 


30% 


23% 


4% 


4. Communication 


33% 


31% 


57% 


38% 


27% 


30% 


30% 


42% 


13% 



Table 3 



Knowledge and Skills Transfer Learning Outcome 







Agree 






Neutral 






Disagree 




Item 


SI 


S2 


S3 


SI 


S2 


S3 


SI 


S2 


S3 


I . Specific Knowledge 


33% 


69% 


74% 


21% 


12% 


17% 


46% 


19% 


8% 


2. Professional Knowledge 


25% 


58% 


70% 


30% 


35% 


17% 


46% 


7% 


13% 


3. Change in Student Learning 


38% 


46% 


52% 


8% 


35% 


26% 


54% 


19% 


22% 


4. Change in Assessment 


38% 


65% 


74% 


30% 


27% 


17% 


33% 


7% 


8% 



Student Perceptions of Electronic Portfolios 

The open-ended questions at the end of the survey asked students to discuss the benefits and challenges of 
electronic portfolios, share their perceptions of electronic portfolios as a method of assessment or tool, and 
if they believed that the electronic portfolio better captured their overall learning and knowledge as 
opposed to other assessment methods. 

In response to the benefits of utilizing electronic portfolios in their classrooms, the field of education and/or 
within the cohort, students believed that their electronic portfolio assisted in keeping them more organized, 
had the advantage of being both portable and paperless, and was easy to retrieve from any location with 
Internet access. Students within the cohort acknowledged the fact that their own students are more 
comfortable with technology and would most likely embrace this concept as a method of assessment. In 
addition, students from the cohort found that over time they were increasing their own technology skills 
while developing their portfolio and felt a sense of pride from sharing their work with their peers. Several 
students are seeing the portfolio as a marketing tool with the ability to search for new jobs and/or further 
their careers in higher education as they believe the portfolio demonstrates their growth as an educator. 

The challenges of electronic portfolios within their classrooms, the field of education, and/or within the 
cohort were overwhelmingly identified as the technological skills needed to develop portfolios and for 
those individuals with a low degree of technological self-efficacy, the frustration experienced within the 
electronic portfolio environment can take the focus away from learning. To utilize electronic portfolios as a 
method of assessment within their own classrooms, technological equity can be an issue for both the school 
and the student regarding availability of resources. In response to utilizing electronic portfolios in her own 
classroom, one student stated, “The usual problems with students, the technology problems of teaching 
students to use it properly and the availablity to all students.” 

Student perceptions of using electronic portfolios as a method of assessment expressed preference of this 
method as opposed to others such as a paper/pencil test, and believed that through their 
reflections/joumaling exercises they were better able to express their ideas and look back on past 
assignments to observe progress made. The portfolio allowed for a broader expression of learning, 
immediate feedback on progress and a more “authentic assessmenf ’ that “personalizes learning” as opposed 
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to memorization and regurgitation of facts. However, students also acknowledged the fact that technology 
presents a barrier, but once overcome, the portfolio has potential to be a good tool for assessment. One 
individual noted, “When I write, when I put together my thoughts, ideas, insights, and reflections, I am 
more cognitively aware of my learning and growth. I believe that even a quick glance at my electronic 
portfolio would better illustrate my professional development than through looking at a typical assessment 
like a test over learned materials.” 

The final open-ended question on the survey asked students if they believed their electronic portfolio better 
captured their learning and knowledge as opposed to other assessment methods. Only one individual 
disagreed with this concept and did not elaborate. Responses overwhelmingly expressed satisfaction with 
electronic portfolios in representing their knowledge and understanding of course content. Students 
specifically targeted the reflections/joumaling exercises as a way to provide a holistic view of their learning 
in that students are able to better clarify ideas and concepts to share with others what has been learned, 
review past projects/courses to compare to later work in order to see how their thought processes have 
evolved and changed, and the freedom to express thoughts and ideas specifically applicable to personal 
situations. One student stated, “I love being assessed via my eportfolio. I would much rather write about 
topics than be tested. I feel I’ve learned more in these three classes about myself as a teacher than I learned 
in four years of college.” Another individual said, “I do believe the e-portfolios better captures overall 
learning than other methods because reflections captures a true sense of learning. There is no pressure of a 
test either which allows us to think about our learning.” 

Lessons Learned 

Lessons learned from the implementation of an electronic portfolio system to document and assess student 
learning range from technological issues and barriers, the time involved for both students and instructors to 
develop and review, and the overall acceptance of electronic portfolios as an assessment method. 

As indicated earlier, students came into the cohort with a variety of technological skills and abilities from 
the novice user to the expert. For students with a lesser degree of technological comfort/self-efficacy, the 
process of learning to use the eportfolio system in addition to developing technological skills can be 
extensive and lead to frustration and anxiety for the individual. A lesson learned from this experience is the 
need to create an electronic portfolio orientation course or workshop focusing on developing students’ 
technological skills and increasing their confidence within the electronic portfolio environment prior to 
using this as a major program assessment tool. It is also important to provide continued technical 
support/assistance if and/or when new skills are required. 

Sharing portfolios was a postive outcome from the implementation; however, it is important to recognize 
that students may not feel comfortable with having their “points of view” published on the Internet. And 
while the reflection process is integral to the portfolio this is an important issue. Electronic portfolio 
systems have the capability to password protect access to portfolios and this should be made an option to 
students especially if part of their grade is dependent upon the reflective exercises. 

Developing and reviewing portfolios is a time consuming process for both the student and instructor. An 
important lesson learned from this implementation is to not only provide ample time to develop and review 
portfolios, but to allow students the freedom to add additional artifacts to demonstrate their understanding 
of specific concepts. The development of comprehensive rubrics can assist in providing students with a 
detailed analysis of their performance and save time for the instructor. 

The implementation of electronic portfolios as an assessment method throughout the program was a 
difficult concept for students and instructors to grasp as it is dynamically different from traditional 
assessment measures. The road traveled by all involved with this process has not always been smooth; 
however, the consistent progress, changes in attitudes overtime, and the ability for both faculty and students 
to see the progress in learning have only solidified the notion that this is a quality measure of learning and 
one that will continue to improve with practice. 
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Introduction 

Rapid change in technology continues to impact education profoundly. The convenience of accessing 
college classrooms from remote locations through web-based technology has made the prospect of attaining a 
college degree a reality for many adult students. This demand has, in turn, increased the need to provide courses 
through web-based formats. By necessity, many faculty have embraced web-based technology in order to meet the 
needs of today's college students. Those who have accepted this new classroom format have had to learn — 
sometimes “on the job” — how to use the classroom management system in order to provide the instruction. 

Yet, despite the early influx of faculty who used the technology, known as compassionate pioneers by Feist 
(2003), the reality of actual use by faculty waned considerably once the novelty of the new toy wore off (Robinson, 
2004). While studies that address the perspectives of students regarding the use of web-based course work may be 
found, very little research has been done that specifically identifies the concerns and attitudes of faculty. What are 
the concerns of faculty who actively participate in web-based instruction? How are these concerns addressed, if at 
all? 

The purpose of this study was to examine the perceptions of faculty about web-based instructional 
technology. The significance of the study was to inform other faculty and administrators about faculty concerns 
regarding teaching in web-based environment and to provide insight into how to address those concerns for optimal 
learning outcomes for students. 



Background 

Faculty who opt to participate in web-based instruction come from all disciplines and with all levels of 
expertise regarding technology use. In addition, the accessibility of training, either in their own institutions or 
through professional associations, ranges from nonexistent to highly formalized. The adoption of any new mode of 
instruction often necessitates change in the individual using the methodology; indeed, as Bailey & Palsha (1992, 
March/ April) state, “the impetus for change often comes from external sources” (p. 226). The peculiarity of online 
environments, in particular, necessitates a change in orientation and in pedagogy. Faculty members new to this 
environment experience stress in navigating the terrain to create effective learning experiences for their students. 
Learning to use the technology while simultaneously trying to use it to develop and deliver the course proves 
daunting for novice faculty (Choi & Park, 2005). 

The literature of faculty concerns about teaching online has focused on factors such as lack of adequate 
infrastructure; training and technical support, administrative support; and effect of teaching online on students’ 
learning outcomes (Kang, 2005; Wilson, 1998; Olcot & Wright, 1995). However, other concerns about teaching in 
this new format are yet to be addressed. Faculty adapting to the teaching of web-based courses go through a process 
that is highly personal, involves developmental growth, both affectively and cognitively, and can be influenced by 
timely interventions directed toward the individuals, the innovation, and/or the context in which the change is taking 
place (Wilson, 1998; Anderson, 1997). Additional research about the introspective perceptions of faculty will 
deepen our understanding of issues regarding the use of technology as an instructional medium. 

One model which examines change in individuals, the Concern-Based Adoption Model (CBAM), applies to 
anyone experiencing change— administrators, policy makers, teachers, parents, and students (Hall & Hord, 1987; 
Hord, Rutherford, Huling- Austin, & Hall 1987). The purpose of CBAM is to determine the concerns that 
professionals may have about an innovation. Concerns are defined as “the composite representation of the feelings, 
preoccupation, thought, and consideration given to a particular issue or task” (Hall, George & Rutherford, 1986, p. 
5). The CBAM holds that people considering and experiencing change evolve in the kinds of questions they ask and 
in their use of the new technology. In general, early questions are more self-oriented: What is it? How will it affect 
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me? When these questions are resolved, then the questions that emerge are more task-oriented: How do I do it? How 
can I use these materials efficiently? How can I organize myself? Why is it taking so much time? Finally, when self 
and task concerns are largely resolved, the individual can focus on impact. Educators ask: Is this change working for 
students? Is there something that will work even better? 

For this research project, the Concems-Based Adoption Model was used as a basis for the development of 
a survey to allow us to examine the concerns that faculty have about the use of course management system and to 
compare their stages of concerns with their levels of use of the various tools in the course management system. 

Methodology 

The purpose of this study was to explore faculty concerns about web-based instruction and the use of web- 
based course management tools. Many concerns emerge when faculty begin to use course management systems and 
the various tools available in that system to design and develop web-based courses. The researchers sought to 
answer the following research questions: 

1 . What tools in a web-based course management system do faculty use? 

2. What types of concerns do faculty have about using the tools of a web-based course management 
system? 

3. Is there a relationship between faculty’s stages of concern and their level of use of the tools of a 
web-based course management system? 

4. If so, do faculty’s discipline/teaching experience/access to training/years of use (familiarity with 
the tools), etc. have effect on that relationship? 

Participants 

The survey was piloted on faculty at Texas A&M University-Texarkana. A&M-Texarkana is a small, 
upper division university in Northeast Texas. The university has a total of 60 full time faculty and offers 
undergraduate and graduate programs in business, education, nursing, behavioral sciences, and liberal arts (including 
history, English, political science and math). The student body has a mean age of 33; while the population of 
traditional students is growing, the current population is still influenced by working adult students who are largely 
place bound and who seek degrees to improve their quality of life within the context of their existing environment. 
Web-based instruction provides a viable alternative access to education for this group of students. Faculty at the 
university were sent an e-mail that introduced the research project, encouraged them to respond, and provided a link 
to the online survey. Announcements were also made in two departmental faculty meetings regarding the survey. 
Due to the self-selecting nature of the survey unintentional bias may have occurred, and the results reflect the 
perceptions of only those who chose to participate. 

Out of the 65 faculty, administrators and instructors invited to participate, 34 (52% response rate) 
completed and submitted the survey. Of the 34 submissions, 30 were completed by faculty, two by administrators, 
and two by those designating themselves as “other.” Sixty-eight percent of the participants are female and 32% are 
male, which indicates a representative sample of the A&M-Texarkana faculty. Twenty-eight indicated they hold 
doctoral degrees and six responded as master’s graduates. When asked how many years they have been involved in 
higher education, nine reported 1-5 years, eight reported 6-10 years and 17 reported 10+ years. Regarding discipline, 
five were from Business, fifteen were from Education, three were from Liberal Arts, four were from Sciences, and 
five indicated “Other.” 

When asked how long they had been involved in web-based instruction, 1 1 indicated “Never”, while 20 
indicated two, three, or four years; 3 indicated that they had five or more years. Of the 34 participants, 10 indicated 
they were novice, 1 5 marked intermediate, and 2 considered themselves advanced. When asked if they had received 
any training in using web-based course management platforms, 22 indicated yes and 12 indicated no. A total of 18 
indicated that they had received training from the local institution. When asked about academic discipline, 12 were 
in education, 3 were in liberal arts, 4 in the sciences, and 5 were “other.” Three people skipped this question. 

Finally, when asked about which course management platform the participant currently uses, 20 indicated WebCT, 2 
indicated Blackboard, 1 uses eCollege, 1 uses WebCT Vista, 12 indicated no use, and 2 indicated “other.” 

Instrumentation 

Stages of concern. According to Anderson, (1997), the CBAM theory “idealizes the Stages of Concern as a 
developmental progression in which teachers implementing a change have concerns of varying intensity across all 
seven stages at different point on the change process” (p. 334). Modeled on the Change Facilitator Stages of 
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Concern Survey (CFSOCG) of the CBAM (Hall, Newlove, George, Rutherford, & Hord, 1991), the first prong of 
the survey for this study consisted of 28 statements addressing the seven Stages of Concern (SoC) (Newlove & Hall, 
1976). This study seeks to determine the current SoC for A&M-Texarkana faculty and explore possible relationships 
to their level of use of the course management system tools. 

The Stages of Concern (SoC) component of CBAM is a framework that describes the attitudes and 
motivations a faculty member might have about the change in practice that occurs at different points during the 
implementation of web-based courses at their institution. At Stage 0, AWARENESS, the faculty member has little 
knowledge about or interest in web-based course management systems or the tools. At Stage 1, 

INFORMATIONAL, the faculty member has a general awareness of the tools and an interest in learning more. 
Faculty concerns at Stage 2, PERSONAL, are focused on the individual’s uncertainty about demands of learning 
about the tools and the overall impact of the change in practice. At stage 3, MANAGEMENT, faculty tend to focus 
on the process and tasks of using the course management system, while, at Stage 4, CONSEQUENCE, faculty 
concerns begin to focus on the impact that their use of course management systems have on student learning 
outcomes. Once they feel more comfortable with their use of the course management system, their concern focus 
moves to Stage 5, COLLABORATION, where they focus on cooperation and coordinating with other faculty 
regarding the use of the tools for teaching and learning. When faculty begin to explore more universal benefits of 
using a new innovation, such as the course management system for web-based teaching and learning, they move into 
Stage 6, REFOCUSING. 

Levels of use. The Levels of Use (LoU) component of CBAM is a framework that describes patterns of 
behavior as faculty progress through the implementation process (from NONUSE to RENEWAL). The second 
prong of the survey consists of a checklist of tools found in most common web-based course management systems 
such as WebCT, Blackboard, FirstClass, Elluminate, among others. The checklist contains response options 
representing the eight levels of use of an innovation of the CBAM. The response options include NON-USE (0), 
ORIENTATION (I), PREPARATION (2), MECHANICAL (3), ROUTINE (4A), REFINEMENT (4B), 
INTEGRATION (5) AND RENEWAL (6). Participants’ responses signify their current level of use of each web- 
based course management tool. 

“As with Stages of Concern, the CBAM Levels of Use schema represents a possible, not a necessary, 
developmental progression in teacher behaviors focusing on the implementation of a specific change in practice” 
(Anderson, 1997, p. 335). Web-based course management systems are multi-functional with a variety of tools, but 
we needed to identify specific changes in practice. Therefore, the checklist consisted of 26 web-based course 
management tools grouped according to functions they serve in a course management system. The categories are 1) 
course content development, 2) course management and usability, and 3) communication/live conferencing. 

Instrument Development 

Instrument development. The development of the survey involved both a collaborative and iterative process 
of four faculty members from A&M University-Texarkana. The research group includes two professors with 
backgrounds in instructional technology, one with a background in marketing, and one in adult education. All four 
participants teach courses online. Once the decision was made to use the CBAM as a model for the instrument, a 
checklist of tools in a typical web-based course managements system was developed and divided into three 
categories: course content development tools, course management and usability tools, and communication and live 
conferencing tools. A matrix was developed addressing each of these tools, and the participants were asked to check 
their level of use by assessing their level of expertise. 

Next, the research group developed a series of open-ended statements using the CBAM stages of concern 
as a guide to determine the respondent’s level of concern. The statements were modeled after statements of concern 
developed by Newlove and Hall in 1998. Seven stages of concern for each category of tools in web-based course 
management systems were addressed. The survey began with 43 statements and was pared down to a total of 27 
items. Development of the questions included an iterative process whereby each of the stages of concern was 
assessed. Each question had the same response options, thus creating a likert scale for participants to self-evaluate 
themselves. 

Following the development of the concern statements, the research group conducted a series of revisions 
for validity. First, the authors completed the survey independent of each other and the initial scoring was examined. 
Revisions were made. Then, the sample instrument was given to a representative audience of university faculty, 
which provided data for the next phase of revision. The authors completed an inter-rater reliability test to check for 
validity. A reliability test was performed on the scores using Cronbach’s Alpha. It was confirmed that the Level of 
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Use questions adequately addressed the three categories of the Course Management System tools with a Cronbach’s 
alpha = .848. The Stages of Concern questions also adequately address the seven stages with a Cronbach’s alpha = 
.929. 



Results 



Level of Use 

Information regarding the level of use was compiled using the Levels of Use categories and then collapsed 
into three broad categories. Category 1 included Non-Users, Orientation, and Participation levels; Category 2 
included Mechanical, Routine, and Refinement; Category 3 included the Integration and Renewal levels. Table 1 
illustrates the breakdown of the 34 participants. 

Table 1 Level of Use Responses (n=34) 



Category 


Levels of Use/ 
Tool 


Non-Use/ 

Orientation/ 

Preparation 


Mechanical/ 

Routine/ 

Refinement 


Integration/ 

Renewal 


#of 

Respondents 
per question 




Page creation tool 


21 


11 


2 


34 




Content Module 


21 


11 


2 


34 




Glossary tool 


29 


5 


0 


34 


'o 


Image Management Tool 


28 


5 


0 


33 


o 


Question Databases 


24 


9 


0 


34 


(D 


Assignment Tool 


23 


8 


3 


34 




Exam Tool 


25 


6 


2 


33 


O 

O 


Calendar 


25 


7 


2 


34 


(D 

in 


Compile Tool 


26 


6 


2 


34 


O 


Syllabus Tool 


22 


8 


3 


33 


O 


Import/Export Course Content 


21 


11 


2 


34 




Change Course Settings 


16 


15 


2 


33 




Reset Course 


21 


10 


2 


33 


(D 


Manage Files 


12 


18 


3 


33 


<D 

&X) 


Share Access to Course 


23 


7 


2 


32 




Manage Students 


16 


14 


2 


32 




Track Students 


17 


13 


3 


33 




Track pages 


21 


8 


3 


33 


o o 


Manage Teaching Assts. 


28 


4 


1 


33 


O H 


Backup Course 


23 


7 


2 


32 




Student Presentation 


25 


4 


4 


33 


o 


Mail 


13 


16 


4 


33 




Chat 


26 


4 


3 


33 


o 

’3 


Interactive Whiteboard 


31 


1 


1 


33 


(D 3 
^ ^ 


Student Home Pages 


27 


5 


1 


33 


=3 S o 

O O o 


Breakout Rooms 


30 


1 


2 


33 


O O H 


Discussion Board 


19 


12 


2 


33 



As shown in Table 1, the page creation tool and the content module are the most used of the course content 
tools while the glossary tool and image management tool are the least used. The assignment tool and syllabus tool 
are the ones that are most likely to be in a state of integration and renewal. Of the course management tools, 
managing files and changing course settings are the most used and managing teaching assistants is the least used, 
probably because the university does not have teaching assistants. Manage files, track students and track pages had 
the highest number of integration and renewal users. 

Mail and discussion board are the most used and interactive whiteboard and breakout rooms are the least 
used of the communication course tools. Student pages and mail had the highest overall number of users in the 
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integration and renewal levels. However, it is interesting to note that, under the stages of concern, non-users 
expressed concern that the communication/live conferencing tools are as effective as in face-to-face setting. 

Stages of Concern 

Of the 34 respondents, 9 were identified as “non-users” by their own self-assessment. To examine the 
stages of concern of those who identified themselves as “users”, the “non-users” were pulled from the data, and the 
responses of the “users” to the Stages of Concern questions were totaled and percentages were tabulated. The 
average percentages are illustrated in Table 2. 

Table 2 Stages of Concern Responses (n=25) 



Stage of Concern 


Not True of Me Now 


Somewhat True of Me 


Very True of Me Now 


“0” Awareness 


1% 


14% 


84% 


“1” Informational 


16% 


40% 


44% 


“2” Personal 


22% 


49% 


28% 


“3” Management 


22% 


43% 


35% 


“4” Consequence 


40% 


35% 


22% 


“5” Collaboration 


26% 


37% 


37% 


“6” Refocusing 


41% 


26% 


33% 



The participants in this study showed high levels of concern at the lower SoC. This indicates that the 
majority of faculty at this institution are in the initial stages of the change process. This is supported by their 
responses to the LoU questions. 

Faculty Concerns by Level of Use 

In order to determine patterns in the data, we grouped the participants based on their Levels of Use and 
then determined the levels of concern for each Stage. Figures 1-3 represent this data for each category of the course 
management system tools. 



Stages of Concern by Level of Use 
Course Content Developement Tools 




□ Awareness 

■ Informational 

□ Personal 

□ Management 

■ Consequence 

■ Collaboration 

■ Refocusing 



Level of Use 



Figure 1 Course Content Development Tools 

Participants representing levels of use from Orientation to Refinement for the content development tools 
appear to have higher concerns at the Informational, Personal, and Management SoC (Fig. 1). These responses could 
be attributed to concerns with the amount of time that is required to develop a course for online production. Because 
of the size of the university, few resources — in terms of support from research assistants, released time for course 
production — are available for faculty. These results are congruent with the outcomes of most CBAM assessments; 
that is, the higher the level of use the higher the level of sophistication of the stages of concern. Those participants at 
the Integration and Renewal levels of use clearly have stages of concern in the Collaboration and Refocusing areas. 

It is also interesting to note that, regardless of the level of use, the Personal stage ranks high. Again, this might be 
explained by the lack of resources and, therefore, the increased amount of time needed to perform the task. 
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Stages of Concern by Level of Use 
Course Management Tools 




Level of Use 



□ Awareness 

■ Informational 

□ Personal 

□ Management 

■ Consequence 

□ Collaboration 
II Refocusing 



Figure 2 Course Management Category 

Participants’ responses on the Course Management category are fully in line with the expected outcomes. 
Individuals in the Orientation; Preparation; and Mechanical, Routine, and Refinement Levels of Use were similar in 
nature. Again, the participants in the Integration and Renewal Level of Use reported much higher stages of concern 
in Management, Consequence, Collaboration, and Refocusing (Fig. 2). 



£ 3 5-1 


Stages of Concern by Level of Use 
Course Communication Tools 
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□ Awareness 

■ Informational 

□ Personal 

□ Management 

■ Consequence 

□ Collaboration 

■ Refocusing 
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Orientation, Mecanical, Integration, 

Preparation Routine, Renew al 

Refinement 

Level of Use 



Figure 3 Course Communication Category 

There was a distinct change in the levels of use for specific tools in both the course management and 
communication categories, but the overall concerns only showed a shift in the communication category. 

Participants’ responses on the Communication category demonstrate a positive relationship where the concerns 
address the higher stages when the user is at a higher level of use (Fig. 3). Considering that only 3 of the participants 
reported five or more years of experience teaching web-based courses, the results suggest that it may take at least 
five years to move through the change process. 



Discussion 

The purpose of this study was to address four research questions. The first question was simply to 
determine what web-based course management tools faculty were currently using. As discussed above, the majority 
of the participants consider themselves to be at the Orientation or Preparation levels of use in all of the categories, 
with a few exceptions. More than fifty percent of those teaching one or more web-based courses demonstrate higher 
levels of use for some of the communication and course management tools. The tools include changing course 
settings, managing students and files, and mail and discussion. Interaction between faculty and students (and 
content) is lifeblood of the learning environment, so these tools must be used for successful teaching and learning in 
web-based courses. It was also noted that fewer faculty members participate at the Mechanical and Routine level of 
use when we look at the course content development tools category. The tools in this category may take longer to 
learn, and many find it faster and easier to transfer documents into the course files rather than create new content 
documents. 

The second question addressed the types of concerns faculty have about using the tools of a web-based 
course management system. While specific concerns were addressed above, similar issues regarding use of personal 
time, lack of administrative support, and effectiveness of student learning were the major issues that faculty at all 
levels of use shared. Those participants in the Integration and Renewal levels were more concerned with the 
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limitations and lack of flexibility of the course management system. This is expected for any new innovation 
because the concern focus moves from personal use of the tools to the purpose and impact of using the tools for 
instructional purposes. 

The third research question initiated this study. The figures 1-3 above clearly illustrate that a relationship 
does exist between the Levels of Use and the Stages of Concern. Those who frequently use more of the tools have 
concerns that involve higher stages of concern. Collaboration with other colleagues as well as interest in modifying, 
developing, or investigating new technologies became a higher priority than the lesser stages of concern. It was 
interesting to note that the participants in this study that have more experience (Integration and Renewal LoU) 
continue to have high personal concern in the course content category. Future research should be conducted to 
determine if this is unique to this pilot institution or if personal concerns about the development of course content in 
web-based teaching and learning is a wide spread phenomenon. Because the faculty at this LoU have five or more 
years of experience with teaching web-based courses, this should provide data that supports the need for ongoing 
faculty training and support. 

The fourth research question — whether a faculty member’s discipline, teaching experience, access to 
training, years of use, or other factors have an effect on that relationship — could not be addressed with such a small 
sample. However, this research project does validate the instrument as a useful measure to address faculty concerns. 
Future research should be conducted using the instrument with a larger sampling from multiple educational settings. 

In conclusion, it was determined that concerns as a whole increased as the level of use increased. This fact 
supports the validity of the instrument used because the CBAM conceptual model identifies stages of concern based 
upon the interaction of the participants with the innovation. This study also supports the premise of the 1999 PT3 
grant proposal. Shared Visions, that “long-term support is key to teacher change” (Hall, Fisher, Musanti, Halquist, 
Magnuson & Simmons-Klarer, 2002, p. 4). The need for strong support from administration with any educational 
change has been historically well-documented (Goodlad, 1968; Lieberman, 1973). This study also supports other 
studies which report the importance of adequate training for instructors and students (Storey, Phillips, Maczewski & 
Wang, 2002). Although the concerns may change, all levels of users have valid concerns that should be addressed as 
all stakeholders move through the change process. Many universities have basic training for use of web-based 
course management systems, but future studies are needed to determine how to address the concerns of faculty after 
they gain experience with the tools. 
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Abstract 

Moon (1999) proposed that it is through reflection that we make meaning of new information and the environment 
and advance from surface to deep learning. The ability to thinking reflectively has been advocated as one of the 
educational goals in schools and colleges (Dewey, 1998). However, because of its complex and elusive nature, 
researchers and practitioners have found it difficult to engage students in this process. Various strategies have been 
recommended for encouraging reflection, such as portfolios(Wade & Yarbrough, 1996), socratic questioning, 
journal writing. Interpersonal Process Recall (IPR), and reflecting teams (Griffith & Frieden, 2000). In addition, 
peer feedback was found to promote deep thinking and critical thinking. This study aims to investigate the effects of 
peer feedback for journaling on college students’ reflective thinking skills. Forty-four first- and second- year 
undergraduate students in a northeastern land-grant university participated in the study. However, only 27 students 
kept journals consistently. This study employed an experimental design with one control group and one treatment 
group. Two journals were sampled from the first half of journal entries and the second half for each student and a 
repeated measure one-way ANOVA was performed to analyze the data. Statistically significant results were found 
that students’ reflective thinking level increased over time but peer feedback affected negatively on students’ 
reflective thinking skills demonstrated in the journals. The result of the study indicated that the use of reflective 
journals promoted students’ reflection as shown by previous studies. However, peer feedback’s adverse affect on 
their reflection suggested more careful use in the future. 



Introduction 



Role of Reflection in Learning 

Based on his observations of his own thinking process and of others’, Dewey (1933; Dewey, 1998) made a 
seminal contribution by giving a detailed account of how we think. Building on Dewey’s work, many other theorists 
continued to propose their own models (e.g. Kolb’s (1984), Schon’s (1983), Mezirow’s (1991), and King and 
Kitchener’s (1994)). Moon (1999) defined reflection as “a mental process with purpose and/or outcome in which 
manipulation of meaning is applied to relatively complicated or unstructured ideas in learning or to problems for 
which there is no obvious solution” (pi 55). In order to illustrate the role of reflection in learning, she proposed a 
hypothetical model, called “a map of learning,” that synthesized the theories of cognitive structure, Piaget’s theory 
of assimilation and accommodation, stages of learning (similar to information processing theory), deep/surface 
learning, and best possible representation of learning (BPR): 

Moon identifies learning as a cycle of assimilating information, building cognitive structures, and 
accommodating new information and cognitive components. Each cycle starts from people taking notice of new 
information, then making sense, making meaning, and working with meaning until transformative learning. Each 
stage of learning results in some amount of upgrading of the BPRs in the learner. “It has been suggested that stage of 
learning reached at a given time determines the maximum (best possible) quality of representation of that learning.” 
(Moon, 1999 pl47). 

Scholars seem to agree on a pyramid-shaped model of reflective thinking, which divides reflective thinking 
into three modes or levels: technical, contextual and dialectical (Taggart & Wilson, 1998). Mezirow (1991) 
differentiated three categories of reflective thinking: content, process and premise reflection. The common belief of 
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the previously cited models is that the degree of reflection is a function of how much of the schema or cognitive 
structure is used or changed. When past experience only flashes through the mind, the learner is not involved in deep 
levels of reflection and the representation of the thinking process (if there’s any) will be mostly descriptive. 

Doubtful situations trigger reflection (Dewey, 1933), and we can hypothesize that reflection serves to accommodate 
new information and minimize the “disequilibrium” in learners. During the process of examining past experiences, if 
the learner finds a problem or doubt, his or her thinking deepens. Thinking can be further more reflective when the 
learner begins to contextualize thinking to find the cause and effect of the situation. If the learner questions his or 
her premise, begins to tolerate individual differences, outgrows egocentrism and moves to exocentrism, and 
broadens thinking into a larger (such as sociopolitical) context, his/her schema is used to a great extent (Moon, 

1999). This idea of different levels of reflection is in harmony with Moon’s model since she stated that extensive 
reflection by the learner will not only make meaning but modify the cognitive structure so that his/her schema is 
used or changed to a large extent. 

Since reflection is an effortful action, previous research has indicated that students find it difficult to 
engage in reflection over extended periods of time without external support (Harri-Augstein & Thomas, 1991). It 
has been found that most college students are involved in “quasi-reflective” thinking as their reflections usually stop 
at the lower level (P. M. King & Kitchener, 1994). Various strategies have been recommended for encouraging 
reflection. Among them, journal writing was identified as an effective way to promote students’ reflective thinking 
skills because it offered a means by which students can externalize their reasoning and reflections on experiences 
(Shekel & Trimmer, 1994) and then reframe experiences within the learning context (Andrusyszyn & Davie, 1997). 
Journaling has become a popular technique that is both found to promote reflective thinking skill (Griffith & Frieden, 
2000; Hiemstra, 2001; Jasper, 1999; Kerka, 2002; Keys, 1999; Priya & Xie, 2005; Spalding & Wilson, 2002) and 
used to assess it because journals provide “evidence” of whether or how reflective thinking skills are used (Boumer, 
2003; Wood & Lynch, 1998). 

Although some research has been conducted in this area, no specific research has investigated the use of 
additional strategies to help scaffold reflection through journaling. For example. Moon suggests that working with 
others can facilitate reflection. And Bond (1999) explicitly mentioned that working with peers instead of someone 
who were presumably “superior” such as mentors or teachers can help reflection. 

Peer -based Learning 

A “critical friend” is said to promote reflective thinking skills. Moon (1999) illustrated as follows: 

“working with others can facilitate learners to reflect and can deepen and broaden the quality of the 
reflection so long as all the learners are engaged in the process. Another person can provide the free 
attention that facilitates reflection, ask challenging questions, notice and challenge blocks and emotional 
barriers in reflection.” (pi 72). 

If the critical friend is involved in another person’s relatively individualized journaling process by 
providing constructive feedback, both parties will have opportunities of seeing different perspectives, which may 
influence or fundamentally change the way they assimilate and accommodate information. No two persons have had 
the exact experiences; therefore their cognitive structures are expected to be different. When two people’s making- 
and working-with-meaning processes are so closely intertwined, they may feel the need to modify their cognitive 
structure not only to accommodate new information but also to tolerate individual differences. If the two people start 
with different values or worldviews, it is possible for an overhaul of the structure to take place, which is called 
“transformation” by Mezirow (1991). 

Bond (1999) compared peer learning with mentoring on the possibility of engaging in reflective thinking, 
commenting “there are increased possibilities for participants to engage in reflection and exploration of ideas when 
they are not in the presence of someone designated as a ‘teacher’”. It was found argumentation facilitates reflection. 
For instance, Blumenfeld, Marx, Soloway, & Krajcik (1996) stated that discourse in online forums encourages 
“delayed reflection”. Eisen (2001) studied the role of peer based learning in professional development from the lens 
of transformative learning defined by Mezirow. She found that “relationally based activities” including peer 
dialogue and feedback sparked individual and joint reflection. Findings in these works were mostly observations or 
reflections researchers made after the experiments were completed. Empirical studies are lacking which directly 
probe into the role of peers on learners’ reflective thinking process. 
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Research Questions 



The current study aims to look into the effect of peer feedback for journaling on college students’ reflective 
thinking skills. The research questions are: 

■ Will students who give and receive peer feedback on their journals exhibit higher levels of reflection 
than those who do not give or receive such feedback? (the null hypothesis is: student who give and 
receive peer feedback on their journals will not exhibit higher levels of reflection than those who do 
not give or receive such feedback and the alternative hypothesis is: student who give and receive peer 
feedback on their journals will exhibit higher levels of reflection than those who do not give or receive 
such feedback) 

■ Does the students’ reflective thinking level differ over time and how? 

Method 



Participants 

Two sessions of the same class with about 70 undergraduate students at a northeastern land-grant university 
were contacted for the study. 44 students signed up for the study but only 27 of them kept journals for the entire 
study span. The data of the rest of the students were deleted from this study. 

Procedure 

Each student in this class was randomly assigned to two groups: control and treatment groups. The control 
group kept journaling for one semester without peer or instructor input or feedback. For the treatment group, 
students were paired and they kept journals for a month as well as responding to their paired peer’s journals. A peer- 
feedback guide was given out at the beginning of the study. All groups were taught by the same instructor and the 
in-class activities were same for all groups. For both groups, the cognitive structure/guideline were given out to the 
students in the “structure” groups at an information session at the beginning of the semester, detailing the five 
phases of thinking proposed by Dewey (1933). Students were asked to write at least one journal every week. Two 
samples were randomly collected from each student’s journals, one from the first half semester, the other from the 
second half 

The “coding scheme of reflective process” was used to score the reflective journals (Frances et al., 1995). It 
was chosen because it has 6 levels and allows for more gradation in scoring. Two raters coded the journals. After 
reaching 100 percent agreement on scoring the first 10 blog samples, raters double-blindly scored the reflective 
journals of the rest sampled journals. The interrater reliability was 83%. The average score of both raters’ grading is 
used for data analysis. 



Result 

A repeated measure ANOVA indicated no interaction effect between treatment and time. For the control 
group who were not involved in peer feedback, students’ mean score on reflection increased by 1.04 point after a 
semester of keeping journals; however, for the treatment group who received and wrote feedback, students’ score 
increased by .93 point over time. 

Table 1 shows the mean reflection scores for both groups in two samplings and the differences of reflection 
scores for both groups over time. Figure 1 shows no observed interaction effect between the treatment and time. 



T abl e 1 Mean reflection scores and standard deviation for two samplings and their differences for both gr oups 



Group 


N 


First sample 


Second sample 


Difference 


Control 


13 


2.34 


3.38 


1.04 






(.69) 


(1.21) 




Treatment 


14 


1.78 


2.71 


.93 






(1.01) 


(.96) 





Note: Standard deviation in parenthesis 
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Figure 1 No observed interaction effect between treatment and time 




group 
— 0 
1 



The following table shows the analysis of variance for the relationship of time and group for their reflective 
thinking score. 



Source of Variation 


SS 


df 


p- Value 


Model 

Time 


13.04 


1 


.000* 


Group 


5.10 


1 


.028** 


Time*Group 


4.07E-02 


1 


.809 



Note: * statistically significant at the .001 level; ** statistically significant at the .05 level 

The interaction effect between time and group is not statistically significant. This means that in the 
population, the effect of group (feedback or non-feedback) on students' reflective thinking scores won't be different 
for the first half or the second half of the semester. And also, the effect of time on students' reflective thinking scores 
won't be different for students who are sending and receiving feedback or not sending or receiving feedback. 





Mean 


Std. Error 


First-half semester 


2.07 


.168 


Second-half semester 


3.05 


.209 



Note: F=19.09, p=.000 

The main effect of time on students' reflective thinking scores is statistically significant at .001 level. In the 
population, as students keep blogging, students tend to be more reflective as time passes by (3.05 vs. 2.07). 





Mean 


N 


Feedback group 


2.25 


13 


Non-feedback group 


2.87 


14 



Note: p=.028 for one-tailed test 

The main effect of different feedback group on students' reflective thinking scores is statistically significant 
at .05 level for the one-tailed test. In the population, students who send or receive feedback on their weblogs tend to 
be less reflective than those who do not send or receive feedback (2.25 vs. 2.87). 

For the first research question, the null hypothesis is retained (student who give and receive peer feedback 
on their journals will not exhibit higher levels of reflection than those who do not give or receive such feedback). In 
fact, the results showed that contrary to the alternative hypothesis, student who give and receive peer feedback on 
their journals exhibited lower levels of reflection than those who do not give or receive such feedback and this 
difference was statistically significant. 
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As for the second research question, it was found that if students kept journaling, their reflective thinking 
level increased over time— the exhibited reflective thinking skill of students increased equally for both groups from 
the first to the second half of the semester. 



Discussion 

The present study investigated the extent to which using peer feedback on college students’ journaling 
could change their reflective thinking skills. The result showed all students improved in their reflective thinking 
skills as time passed by. However, the students who were involved in solitary journaling demonstrated higher level 
of reflection consistently over time than those who sent and received feedback. The differences were statistically 
significant. 

As many researchers have pointed out that when students are required to journal for their coursework or 
field of study, the link between learning experiences and reflective activity could be strengthened because “a 
specific allocation of time which can be used for reflection” is incorporated (Bond, Keogh, & Walker, 1985). Britton 
(1978) identified the role of writing in the reflecting process as allowing students to switch roles between 
participants and spectators of their own thinking. When students are writing, in order to produce an articulate 
statement they need to first construct ideas in their mind — the meaning making process in Moon’s model. Then 
when students pause and become readers of their own writing, they have another chance of speculate on these ideas 
and test their viability according to their existing schema. In order to articulate their ideas, students assume roles of 
participants and readers so that the “learning and the representation of the original learning both occur” at any time 
in Moon’s model. Since the representation of the learning is available to the learners, the workload in their working 
memory will be greatly reduced. Therefore, students can better engage in their learning — making and working with 
meaning. The findings of the study confirmed that if students are constantly engaged in journaling activities, their 
reflective thinking level demonstrated by their journal entries would increase over time. 

Contrary to the prediction of previous research findings, peer feedback did not promote students’ reflective 
thinking skills when combined with journaling. Although students were randomly assigned to different feedback 
groups, the students in the feedback group constantly showed lower reflective thinking level than those who were 
journaling in a secluded manner. Peer feedback seemed to have counteracted the effect of journaling. There could be 
two reasons that had caused the problem: first, journaling is a self-introspective process, when people are journaling, 
they can be distracted or interfered by the fact that their writings will be looked at by some other people and they 
might have refrained from journaling something they thought other people wouldn’t understand or maybe laugh at. 
Research found that individuals wrote something “presentable” and “sense-making” when they knew that there was 
a good chance that other people would read them (Xie & Sharma, 2004). They deliberately avoided “babbling” what 
was on their mind. Instead, they adopted a more conservative approach to journal. The second reason was that it 
seemed students did not engage in a meaningful feedback activity as was found by a closer look at their comments 
to each other. The students in the treatment group were paired. It was found that if one of the pair didn’t demonstrate 
higher level of reflective thinking in their journals, it was very likely that the other won’t engage in higher level of 
reflective thinking either. It seems to be a reciprocal adverse effect on their attitude toward reflection from peers. 
Previous researchers (Slavin, 1990, 1995) pointed out that peer feedback on journaling should be moderated to 
reduce off-track and passive behaviors in interactive discourse, and that structured protocols should be used while 
ensuring opportunities for equal participation and a “trusting and non-threatening relationship” among peers (Risen, 
2001). In addition, structured peer learning improved academic achievement and boosted student attitudes (A. King, 
2002; Zhang, 2004). Bonk and his colleagues identified four “key action areas” for instructors’ moderation (Bonk, 
Kirkley, Hara, & Dennen, 2001). They consist of pedagogical, social, managerial and technical actions. The 
pedagogical role requires the instructor to e.g. ask questions, provide feedback, probe etc. The social role involves 
creating a friendly environment and offering affective support. The managerial actions can include designing and 
coordinate assignment and overseeing tasks and course structure and requirements. The technical role includes 
actions such as assisting the use of technology. 

This study has found that beyond instructors’ moderation on the feedback, it could also be helpful if 
scaffolds could have been provided by the instructor about how to find topics to journal about for the first-year 
college students. The researcher found that some students did not understand what reflection was and how this 
strategy could help them in their study. Although students were given a short information session about how to 
reflect, most students were unsure how to implement their reflection. As a result, many students either recounted 
what they read or learned in class or commented on the way the instructor conducted the class. Even as their 
reflective thinking scores increased over time, their average score by the end of the semester was only 3.05 out of 6 
(the highest possible score of reflection with the coding scheme used in this study). It was indicated in research that 
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self-selected topics in journaling made graduate students question the learning content more deeply because students 
are perforce engaged in finding the trigger for a “doubtful” situation while trying to identify a topic to write about 
(Xie & Sharma, 2004). However, a lot of lower-class college students are not cognitively ready to find the trigger by 
themselves without any help from the instructor. It is recommended that in future studies scaffolds should be offered 
that focus students on the key points of the new learning and contain prompts for them to mentally scan existing 
cognitive structure in an attempt to find conflicting ideas. 

The ability to think reflectively is germane to learning in that it makes learning meaningful. Efforts should 
be intensified to identify cost effective and feasible ways to improve this indispensable skill. The present study can 
serve to provide some insights about how college students make uses of journals and peer feedback. More research 
is needed to advance our knowledge of finding scaffolding strategies for both peer feedback and journaling to 
promote reflective thinking skills. 
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Abstract 

This case study analyzes the cognitive process underlying a teenager’s Sokoban problem solving within the 
framework of major learning theories. The effects of metacognitive inquiries and self-correction on latency in 
completing the game were examined. A major finding was that the gamer employed both forward- and backward- 
working strategies and tended to use a start-all-over-again strategy when faced with an impasse. Implications that 
support a game-based problem solving approach were also discussed. 

Introduction 

What goes on a gamer’s mind working on a play? 

Game-playing was suggested to enhance young learners’ critical thinking and problem-solving skills (Katz, 
2000; Prensky, 2000). Over the past years, a great deal of attention has been paid to games as a powerful learning 
mechanism which goes beyond just a delivery context of content (Squire, 2005). However, there appears to be lack 
of research focusing on how game-playing boosts learners’ critical thinking or problem solving skills. From a 
pedagogical perspective, it is critical to understand how learning occurs while young learners are engaged in 
problem solving given in a context of game play in order to provide them more timely and effective support for their 
learning. Consequently, the knowledge gained through studies about young learners’ thinking via game play over a 
computer may allow us to anticipate potential obstacles that individuals may face in the problem solving process, 
thus making it possible to prescribe timely support for solving such problems. 

Moreover, game-playing in a computerized setting is likely to demand different cognitive and 
metacognitive controls from the conventional, paper-pencil, problem solving approaches. Tracing a learner’s 
thinking as well as learning processes in a computer game format of problem solving may disclose many unique 
aspects of cognition. From such a viewpoint, this study was intended to examine a 13-year-old boy’s Sokoban 
computer game playing. The focus was on following the participant’s flow of thinking while he was working on the 
game. 

Figure 1. The Sokoban Game 




In Sokoban, a player acts as a person who must push a number of crates through passageways to a final 
destination area. The goal is to use minimum number of pushes. The player can only push one single crate at a time 
in one direction, and is not able to pull crates. The gamer can use Undo and Restart buttons when desired. Inevitably, 
the player faces an impasse; there is no way out. The game has a total of 20 levels, each level has an ascending level 
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of complexity. Given the design and rules required, Sokoban provides an ideal training context for self-directed 
problem solving. Sokoban is available for free at www.pimpemel.com/sokoban/sokoban.html . 

Studies have shown that a learner’s metacognitive and adaptive self-correction of cognitive strategies enhance self- 
regulated learning (Butler & Winne, 1995; Zimmerman, 1989, 1994). Based on this idea, this study also examined 
how the learner’s guided metacognitive visualizing of his own thinking process impacted his learning process. 
Metacognitive visualization referred to encouraging a learner to externalize orally and graphically his/her use of 
strategies or mles which led to either a success or failure and adaptively providing corrective feedback. This process 
was believed to aid the participant’s self-regulated monitoring of his learning process. Implications for support of 
young learners’ developing problem solving ability in a game-based learning environment were also explored. 

Theoretical framework 



Problem Solving Stages 

Problem solving generally undergoes the following four stages: representation of the problem, 
searching the problem space, evaluation of the solution, and application (Gagne, Yekovich, & Yekovich, 1993). 
Problem representation in the first stage includes defining the problem and exploring the required conditions of the 
given problem. Searching the problem space, the second stage, refers to exploring the ways that lead to a solution to 
the problem. In this stage, the problem solver may attempt to find a clue for the solution by trying to recall prior 
experiences of problem solving that were similar to, or associated to some extent with, the current problem. In the 
evaluation stage, the problem solver judges a hypothesized solution to the criterion of the goal or solution. The 
problem solver will also evaluate whether the required conditions are met by the solution he hypothesizes. In the 
application stage, the problem solver applies prior learning to a novel situation of problem solving. Although 
problem solving is generally described as a sequence of stages, problem solving would not necessarily always occur 
in the same sequence(Gagne, Yekovich, & Yekovich, 1993). It is assumed that a problem solver working on 
Sokoban goes through all stages of problem solving but the sequence would not be linear. 

Procedural Knowledge Processing 

Cognitive psychologists claim that the goal and sub-goal relations among productions (If-Then) promote 
the control of cognitive flow from one production to another within a given production system (Gagne, Yekovich, & 
Yekovich, 1993). 

It appears to be almost impossible to get direct observation on how a person processes procedural 
knowledge based on any data obtained from the traditional paper-pencil assessment tools. In a problem solving 
scenario utilizing a game format, if the interaction process between the problem solver and a problem is recorded, 
the building process of procedural knowledge can be examined. In addition, the problem structure of the game may 
demand multiple layers of goal setting and decision making through the course of exploring a solution. The learner 
will not only attempt to recall the existing procedural knowledge to relate to the current problem but also create 
procedural knowledge that serves to identify a solution to the novel problem. In this analysis, the focus was the 
examination of the ways in which the gamer formulated goals and sub-goals during the game-play. Considering that 
Sokoban problem solving does not provide a specific goal state, the process of goal setting and goal-directed 
decision making are totally at the player’s discretion. 

Problem Solving Strategies 

There are general strategies to guide exploring a solution in problem solving such as trial and error, means- 
ends analysis, and reasoning by analogy. Recent research findings have suggested that domain-specific strategies are 
also crucial for success in problem solving. Domain-specific knowledge helps construct a schema that “allows 
problem solvers to recognize a problem state as belonging to a particular category of problem states that normally 
requires particular moves” (Sweller, 1988, p. 259). Thus, possessing sophisticated schemas is an essential criterion 
on which an expert is distinguished from a non-expert. According to Sweller (1988), experts tend to employ more 
forward-working strategies than back-ward working (means-ends) strategies, which is attributable to their well- 
established knowledge structure in the form of schemas. 

Sweller and Levine (1982) found that non-specific goals permitted more rapid learning of essential 
structural characteristics than providing a conventional goal. It was argued that in the conventional goal-directed 
learning settings the students had difficulty inducing the relevant rule without implicit or explicit additional 
information provided. According to Sweller (1988), learners use heuristics, such as means-ends strategies, in 
conventional goal-directed learning settings. The use of heuristics, consequently, interferes with “learning essential 
aspects of a problem’s structure” (p. 4). Sweller also empirically showed that problem solving with a specific goal 
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demands relatively large amounts of cognitive resources, thus resulting in additional expenditures of cognitive 
processing capacity required for constructing schemas. 

From Sweller’s perspective, a Sokoban problem solver was expected to discover both goals and methods 
for the play, relying mostly on a forward-working strategy because Sokoban does not have any specific goal 
statement. Considering that the participant in this study was new to Sokoban problem solving, he was anticipated to 
work forward without being controlled by a schema, simply exploring the problem space in order to see what moves 
were possible. (Sweller, 1988). 

Learners’ uses of cognitive strategies are likely to be influenced by the structure of a problem as well. 
Sokoban has a unique problem, structure and context; it has simple game rules and is played on a computer screen. 
As a computer game, Sokoban has a feature that the conventional paper-pencil problem solving does not have. In 
Sokoban a move made acts as a quick, visual clue for the following moves. For example, a wrong move would cause 
immediate, negative feedback to the problem solver because the wrong move inhibits any further action, which can 
inform the problem solver that his action intended was wrong. Sokoban is designed so that the problem solver 
inevitably arrives at many dead-ends. In this study it is also of interest to see what kinds of strategies benefit the 
gamer when put in no-way-out situations. 

Metacogniton and Metacognitive Visualizing 

According to Flavell (1979), metacognition is “knowledge and cognition about cognitive phenomenon”(p. 
906) and involves deliberate, intentional, and goal-directed mental operations. Metacognition comprises two related 
aspects: understanding what kind of skills, strategies, and resources a task requires and knowing how and when to 
use the skills or strategies to ensure the task is successfully completed (Schunk, 2004). Studies indicate that 
regulatory strategies to accomplish specific learning goals are positively related to self-regulating ability (e.g. 
Printrich, 2000; Zimmerman, 2000; Schunk, 2001). 

Problem solving such as Sokoban requires a relatively high level of self-regulated ability in monitoring 
progress, predicting outcomes, and evaluating the effectiveness of strategies or skills. Thus it was considered that 
through scaffolding, the participant visualizes ways to regulate the learning process or uses cognitive strategies to 
enhance his self-regulated problem solving skills. 



Method 



Participant 

The participant in this case study was a 13-year-old boy. From prior conversation with the participant, he 
indicated that his weekly computer game playing totaled approximately five hours per week, mostly for 
entertainment purposes. The participant reported that he has never played Sokoban prior to his participation in this 
study. When he was questioned about other activities with his computer, he responded that he also uses his computer 
to complete homework assignments and class projects several times per week. Based on this information, it was 
regarded that the participant was relatively familiar with playing computer games, mostly entertainment types of 
games, and had fair experience in using a computer for his academic work. 

About the Method 

For the analysis of the thinking process of playing Sokoban, the classical ‘thinking aloud’ approach was 
used along with video- recorded monitoring. In applying these methods, the participant was asked to state aloud 
whatever ideas came to mind while playing the game. 

The traditional method of ‘thinking aloud’ has a distinct advantage in that it allows researchers to obtain 
information not only about the final result of problem solving but also about the learner’s thinking process which 
underlies the game playing. However this method has a limitation in that it demands dual cognitive duties from the 
participant. The participant has to work on the problem while simultaneously recalling from his memory what he has 
done in order to describe it orally. During this process, it’s probable that the limited capacity of our short-term 
memory would not allow the participants to remember all ideas or thoughts that would come up while interacting 
with a problem. 

However this limitation of the ‘think aloud’ method can be canceled out if the overall performance of the 
game player is recorded. On top of that, an audio-visual recording of the overall process of problem solving would 
be useful for timely locating the very points over the course of problem solving where the gamer is in need of any 
cognitive support. The computer screen and the learner’s voice while playing were recorded. The recorded voice 
was transcribed for analysis. 
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Procedure 

Two rounds of game plays were conducted. In order to examine the effects of scaffolding metacognitive 
visualization and guided self-correction of misconception or inefficient use of skills, the participant was intervened 
after the first session of game-playing. In the initial play, the player was directed to state aloud whatever ideas or 
thoughts come up with while playing. There was no restriction to the number of game levels that the participant can 
attempt up to a maximum of 20 levels. On the following day, the participant engaged in the second play. 

Prior to the second game-plying, the participant was guided to visualize his thinking based on the recording 
of the initial game-playing and accordingly asked to self-correct misconceptions or inefficient use of strategies. The 
participant reviewed his recorded game-playing from the previous day, and was asked to respond to the following 
question: “Tell me some rules or strategies that you found useful for the game from the previous play”. This 
question was intended to allow the learner to recall any conditional knowledge that he learned from the prior game- 
playing. It was assumed that answering this question would be possible when the participant figured out what 
conditions signify a success or a failure. 

Then, the participant was also asked to illustrate patterns that might lead to a dead end, which was intended 
to help the learner regulate his own use of conditional knowledge to solve the problem. He was then asked to draw 
the patterns that led to a dead end in the previous play. Next, the researcher and the participant went back to the 
Sokoban screen and the participant was asked to make up situations that might be problematic. Finally, the 
participant was questioned as to why the situations that he created were puzzling to him. This question was intended 
to allow the participant to probe any misconceptions and use of inefficient strategies. In response to the answers 
from the participant, the researcher manipulated a few examples and non-examples of solution paths over the 
computer. 

The second session of play was carried out in a condition identical to the initial play session with the 
exception of allowing an upper cap of the fourth level in order to equate the number of levels attempted between two 
game sessions. The difference on the completion level between two game sessions was examined. The details of 
study procedures are seen in Table 1 below. 

Table 1 . Details of the Study Design 





Session 1 


Session 2 


Interventions 


No intervention. 


Before beginning the game-playing, the 
participant a) was asked by the question, 
“Tell me some rules or strategies that you 
found useful for the game from the 
previous play 

and b) was requested to “Draw the patterns 
that led to a dead end in the previous play”. 

After reviewing of the recorded, previous 
play, the participant was guided by an oral 
discussion and encouraged to self-correct 
his misconception or use of inefficient 
strategies. 


Directions 

provided 


The participant was asked to say aloud 
whatever ideas or thoughts come up 
with. 


The participant was asked to say aloud 
whatever ideas or thoughts come up with. 


Levels 


No upper limit 


The highest level was the fourth to match 
the final session 



* Note: the second playing was carried out on the following day of the initial game-playing. 
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Results 



Effect of Metacognitive Visualizing 

In the initial play, the player attempted the first four out of the total 20 levels for 24 minutes. The player 
completed only the first level out of the four attempted. On the other hand, in the second session of playing, the 
player attempted the first four levels for 28 minutes and completed all four levels attempted; specifically, in the first 
session, the player completed the first level, gave up the second level after several attempts, almost skipped the third 
level, and almost completed the fourth level. In contrast, in the second session of playing, the player was successful 
in completing all four levels within less time than that spent in the first session. 



Table 2. Summary of time spent and completion levels of two game plays. 





Session 1 (day 1) 


Session 2 (day 2) 


Total spent time 


24 minutes 


28 minutes 


Levels 

attempted 


Level 1 to 4. 


Level 1 to 4 


Levels 

completed 


Only the first level 


All 4 levels attempted 



Prior to beginning the second session of playing, the player responded to two requests. First, the participant 
was asked, “Tell me some rules or strategies that you found for the game from the previous trial.” The player 
responded as the followings: 1) “Don’t place the box in a comer”; 2) “Don’t place two boxes side by side”; 3) Find 
the “four-way intersection”; and 4) “Think ahead”. These responses showed that although the player ended up 
completing only one level of the game in the initial game-playing, he was able to discover the cmcial mles and 
strategies for playing Sokoban in that session. . When asked, “Draw the patterns that led to a dead end in the 
previous trial.”, the player graphically described multiple situations that he figured out as being problematic in his 
play. The descriptions showed that the player discerned most of the potential problematic situations. One 
misunderstanding concerned play patterns or mles that led to a dead end. The player might consider that ‘placing 
two boxes side by side’ is consistently problematic in Sokoban problem solving based on his statement, “Don’t 
place two boxes side by side” as a mle for the play. Accordingly, a few situations of examples and non-examples of 
“boxes placed side by side” that leads to a no-way-out situation were modeled onto the Sokoban (see Figure 1). 

Figure 1. Illustration of Examples and Non-examples of the side by side situation leading to a dead end 




* Although both picture 2 and 3 describe a ‘side by side’ situation, only picture 2 leads to a dead 
end. This was articulated to the participant to help his forming a correct concept about the mle. 



394 







Then, the participant was asked if all of those situations were likely to end up with a dead end. The learner 
could arrive at the correct response after actually performing a few moves for the situations on the computer. During 
the discussion, the participant indicated that anticipating a few moves ahead was a useful strategy to produce a 
successful result of the playing. 

Based on the analyses of the video-recorded data, some commonalities and differences were examined 
between the two game sessions. Commonalities included: 1) At times, the player reminded himself of the rule of 
‘minimum moves’ saying “Since I have to make as minimum moves as possible”; 2) The player arranged objects in 
a way to accommodate later moves, a strategy formed based on his prior, general game-playing experiences; 3) The 
player often employed a “start all over again” strategy when stuck; and 4) The player often applied what he learned 
from the previous level to the next level (e.g., the player attempted take a detour at the place where the obstacles 
resembled a four- way intersection in a new level of play). From this observation it was obvious that the player was 
regulating his own playing in order to make his move more efficient. 

On the other hand a few differences were also observed between two play sessions. For example, in the 
second play session: 1) The player used the strategy of seeing a few moves ahead (‘think ahead’, in the participant’s 
words. See Table 3 for details) more frequently; 2) The player spent more time at the beginning of a level and at the 
points of making any change to a move that was already made; and 3) A reduction in the number of dead end 
occurrences were observed. This result indicates that, with increased game playing experience, the player relied 
more on the forward- working; additionally, his use of skills became more automatic, in which it can be assumed that 
he had formed schemas for playing Sokoban. 

Analysis of the Recorded Game-playing 

Use of General Problem Solving Strategies 

It was found from the two trials of game-playing that the participant used both forward- and backward- 
working strategies of problem solving. In the first game playing session. Level 1, the participant appeared to engage 
in forward- working from time to time, as evidenced by the occurrences in which he reminded himself of the ‘global 
goal’, moving all the objects to the destination location, and the general game rules such as only pushing and 
minimum moves: 

My goal is to get all these boxes into six stacks on the right... I have to get the boxes to the four way 
intersection [the place where the array of the obstacles takes the shape of a four-way intersection on a 
road] ... .1 have to make as minimum moves as possible. . . 

At the same time he repeated trials and errors and often arrived at a no-way-out situation before finding a 
solution path. Backward- working strategies were also observed in instances when, for example, he evaluated 
whether his moves had been working or not based on the rules that he figured out for himself In Sokoban playing, 
such autonomous evaluations of the game player appeared to be promoted by the immediate visual feedback given 
by the consequence of an action, which is clear from the following statement of the player: 

I will try this one. ..lam stuck again. I am not going to try with that box any more. . . I need to try to move 
with something else. . ..Oh I got an idea. . ..If I move one more step, I will be stuck. Oh I got it. . . I messed 
up. I am going to try all over again. 

It was also found that the player figured out by himself essential rules for the game play from the early 
stage based on the following statement, “I have to get the boxes four-way intersection” A four-way interaction is 
located in Table 4, in the area labeled 1. 

As he obtained more skills, he employed more often forward strategies, visioning ahead a set of moves. At 
the start of Level 2 in the second session, he stated, “I have to try to get this one out and then I can come back the 
other way. Then, this box should be in the intersection.” (See the figure in Table 4 below). 

Use of Domain Skills and Strategies 

In Sokoban, as true for many other game plays, the participant was not pre-trained with uses of skills or 
strategies that were useful for the game. The participant was entirely left to discover the rules or skills through his 
game-playing. As the participant played more levels, use of strategies was more often observed, indicating that the 
participant’s level of understanding was increasing with time. The player used strategies such as ‘think-ahead’, 
‘taking a detour’, ‘minimum moves‘, and ‘start all over again’. 

‘Think-ahead,’ as named by the player, was commonly observed in this game play. ‘Think-ahead’ strategy 
was apparent based on the participant’s statement in the second session, in which he stated, “I have to try to get this 
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one out and then I can come back the other way. This box should be in the intersection.” Thinking ahead often 
involved the detour strategy as seen in the table 3. 



Table 3. Elaborations of the participant’s use of the ‘Think-ahead’ strategy 




“This box should be in the 
intersection.” 



“. . .and then I can come back 
the other way.” 



“I have to try to get this one 
out” 



Statements 



Graphic Elaborations (Second Session, at the start of 
Level 2) 



It was clear that the player was intentional in employing the ‘think ahead’ strategy in an effort to make his 
actions more successful. This indicated that he was taking a metacognitive control over his Sokoban game-playing. 
In using this strategy, the player’s oral reporting appeared to influence him. Saying loud what he has on his mind 
while playing the game probably facilitated his planning a few steps in advance and thus reinforced his use of the 
‘think ahead’ strategy because he possibly recognized that the strategy was useful to prevent an incorrect path. 

The player also attempted to make ‘minimum moves’ if possible. From time to time, he reminded himself 
of this rule, saying, “I have to make as minimum moves as possible.” Although this ‘minimum moves’ rule was 
offered as a game rule at the outset, the player appeared to utilize this strategy in an attempt to save his cognitive 
efforts as well. 

The player also often played the game from scratch by pressing the reset button, which action was referred 
to as ‘start all over again” by the player. This ‘start all over again’ was used whenever the participant recognized ‘no 
way out’, which is obvious from the following statement: “If I move one more step, I will be stuck. Oh I got it. . . I 
messed up. I am going to try all over again.” The player’s multiple applications of this strategy over different game 
levels showed that the game player had acquired the pertinent conditional knowledge; that is, knowing how and 
when to use the skill or strategy to ensure optimal performance. 
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Developing Procedural Knowledge 

Analyses of Sokoban game-playing with ‘think-aloud’ strategies provided a snapshot about how a game 
player obtained procedural knowledge. In this study, the problem solver developed procedural knowledge based on 
the game goals that were discovered by the player himself rather than provided externally. In Table 4, the 
development of player procedural knowledge is further elaborated. 




Table 4. Elaboration of production development -related to Table 3 



If the sub-goal is to move the crate with 
the white arrow to the place of 1 and go 
around back to the location of 1 to push 
the target crate into the destination, 
Then I need to get rid of this crate 
(pointed by the green arrow) first. 



If the sub-goal is moving the crate (with the 
white arrow) to the destination area 

Then I have to put the target crate into 
the location of 1 . 

And then I have to go way back (arrow 
2) in order to push the target crate into 
the destination area. 



Forming procedural knowledge was evidenced generally after the player failed in his repeated attempts to 
find a solution. Thus, it was regarded that producing procedural knowledge was also kind of a strategy that the 
player employed in an attempt to recover, an anticipated difficulty in the course of problem solving. Based on video 
recording of game playing, it was obvious that the development of the participant’s procedural knowledge increased 
and became more automatic as the total time engaged in Sokoban playing increased. 

Conclusion 

In this study we analyzed how a young game player learns and uses skills or strategies to play the Sokoban 
computer game. The focus was on the player’s metacognitive control over his thinking processes to complete the 
game. In addition, the effect of the player’s metacognitive monitoring and adaptive self-correction of 
misconceptions or ineffective uses of strategies for the game-play were also examined. 

Based on the comparison of the two sessions of game-playing, it was discovered that the participant 
benefited from monitoring and visualizing his problem solving process and guided self-correction of misconceptions. 
In order to separate the possible practice effect from the intervention, further studies may include a control condition 
including multiple subjects or additional game-playing sessions. These findings imply that helping a problem solver 
monitor his learning and to recognize the efficient and inefficient use of skills or strategies is an effective 
pedagogical method. This study also corroborates the findings of Schunk and Swarts (1993), who suggest that young 
learners need feedback to apply strategies in order to improve their problem solving skills. 

Two trials of game-playing indicated that the participant used both forward- and backward-working 
strategies. The backward- working strategy was also observed within a forward- working strategy. Sweller (1988) 
once categorized forward-working strategies. A schema driven approach used by an expert is a common example of 
a forward-working strategy. Further, forward-working strategies can occur during means-ends analysis. The 
problem structure, which has a non-specific goal, can lead to more forward-working (Sweller, 1988). Therefore, 
forward-working strategies were observed in this study and appeared to be related to the structure of the Sokoban 
play that had a non-specified goal. As expected, it was observed that as the problem solver gained more expertise 
over time, his process of playing became more automatic. This suggested that the player had obtained schemas of 
Sokoban playing. Obtaining schemas of Sokoban game playing were clearly seen in this study, based on the 
participant’s use of procedural knowledge. The participant also used backward- working strategies such as means- 
ends analysis and trials and errors. It is probable that the participant began working forward with the non-specified 
game goal and without prior experience of Sokoban playing, thus simply exploring the problem space in order to see 
what moves were possible (Sweller, 1988), without being controlled by a schema. However, when he encountered a 
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difficulty, he seemed to employ more of the means-ends analysis. This potential relationship between difficulties 
and use of backward-working strategies may be tested in hiture studies of problem solving with computer games. 

It was also observed the player used domain-specific strategies such as ‘think-ahead’, ‘taking a detour’, 
‘minimum moves‘, and start all over again’. He seemed to consciously use such strategies as ways to make his 
moves more successful, which indicated that the player kept on monitoring his metacognitive control over his 
problem solving process. The method of ‘think-aloud’ was utilized to reinforce the player’s use of strategies. The 
demand of speaking about his work processes possibly encouraged the participant to think about a set of moves 
ahead and finally helped the player learn that those strategies were useful. 

Based on the recorded data, the participant was also observed to form procedural knowledge in the event he 
perceived a difficulty in finding a solution. Thus it seemed that the player who was working on Sokoban produced 
procedural knowledge in part as a strategy to overcome the difficulty in problem solving. This aspect also confirmed 
the work of Schunk (2004) that, “young learners are more likely to monitor their activities on tasks of intermediate 
difficulty as apposed to easy tasks”(p. 193). 

As a case study, this study has a few limitations. One limitation is that the results may not be generalized 
with other participants, or other types of computer games. This study includes only one participant; therefore, the 
findings may be only relevant to similar ages or developmental levels; simply put, the findings from this study might 
not be consistent with learners at different levels of cognitive ability, computer experience, or motivation. In future 
studies the effects of such variables may be tested to take into further account learner variables. Second, in this study, 
the Sokoban computer game play involves only one gamer without any specific goal statements built-in. The 
findings of this study may vary with other types of game-playing in terms of the number of players, degree of 
entertainment, type of reward, and difficulty. Futures studies may take into account these task variables and test 
effects of such variables with both Sokoban and other types of plays. In addition the results from this analysis imply 
that Sokoban game-playing can be useful in studying cognitive and metacognitive thinking process. 
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Effects of Direct Manipulation and Animation in Questions and Feedback: 

What’s the Fuss? 



Li Zhu 

United States Naval Academy 

Abstract: This quantitative research investigated the instructional effects of different questioning and feedback 
strategies. Five treatment groups included: no questions and no feedback control group, traditional matching 
questions with Knowledge of Response (KOR) text feedback, traditional matching questions with KOR text plus 
animation-elaborated feedback, direct manipulation matching questions with KOR text feedback, and direct 
manipulation matching questions with KOR text plus animation-elaborated feedback. Participants were 184 college 
student volunteers. The study included an online pretest, followed by the web-based treatments and three online 
criterion tests. Two-way multiple analysis of covariance (MANCOVA) was used to analyze the data. Important 
findings included: (1) traditional matching questions were equally effective as direct manipulation matching 
questions in facilitating factual, conceptual, and rules and principles learning; (2) traditional matching questions 
were more effective in facilitating learning difficult concepts than no questions; and (3) no feedback, KOR text 
feedback only, and KOR text plus animation-elaborated feedback were equally effective in facilitating factual, 
conceptual, and rules and principles learning. 

Questioning and feedback strategies have long been considered very important and effective instructional 
strategies. With the development of computer technologies and new software capabilities, it is now possible to 
provide various innovative, engaging, and interactive questioning and feedback strategies in CBI (Computer-Based 
Instruction). Many instructional designers and educators have been using direct manipulation as a questioning 
strategy and animation as a feedback strategy, however, the effectiveness of these two instructional strategies on 
learning is not yet fully understood. 

Direct manipulation questioning strategy allows learners to directly drag and drop screen objects to answer 
questions. The direct manipulation interface gives the learner the feeling of directness which shortens the distance 
between the learner’s intentions and the physical facilities provided by the system, as well as the feeling of 
engagement (Hutchins, Hollan, & Norman, 1985). Although it seems that direct manipulation questioning strategy is 
very promising and innovative, further research needs to be conducted to find out its instructional effectiveness on 
student learning. 

Animation, because of its unique attributes, is often used as a part of the practice strategies in CBI. When 
used in the most structured practice activities, animations were used as feedback to student answers providing 
reinforcement to correct answers (Richer, 1990). Advances in technologies make it possible to enhance the quality 
and type of feedback generated in response to learner’s actions by integrating animation. However, the effectiveness 
of animation feedback strategy in instruction still needs to be determined. 

Review of the Literature 



Theoretical Framework 

Information processing theory, levels of processing theory, dual coding theory, and generative learning 
theory provide the fundamental theoretical framework for this study. 

Information processing theory explains how people perceive, encode, and retrieve information. Direct 
manipulation matching questions and animation-elaborated feedback that match cognitive processing requirements 
of three levels of learning (factual, conceptual, and rules and principles) were expected to better facilitate the 
rehearsal, encoding, and retrieval processes, and therefore, better facilitate learning. Gagne (1985) identified nine 
processes of information processing and related external instructional events to each process. Using his model of 
information processing as a framework. Wager and Mory (1993) analyzed the different roles that questions and 
feedback may serve in learning at each stage of information processing. Questions and feedback provided in this 
study mainly were expected to serve the functions identified in three stages of information processing in their 
analysis. Table 1 provides a summary of the expected roles of direct manipulation matching questions and 
animation-elaborated feedback during those three stages of information processing. 
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Table 1 

Expected Roles of Direct Manipulation Matching Questions and Animation-Elaborated Feedback in the Three 
Stages of Information Processing 



Stages of Information 
Processing 


Expected Roles of Direct Manipulation 
Matching Questions 


Expected Roles of Animation- 
Elaborated Feedback 


Rehearsal to maintain 


• 


Provides practice on previous learned 


• Provides dynamic visual 


information in STM 




information 


reinforcement to the learner’s 




• 


Helps learners organize, group, and chunk 
information into meaningful units 


response 


Semantic encoding for 


• 


Facilitates and strengthens rapid encoding 


• Validates integration 


storage in LTM 


• 


Helps learners integrate new knowledge into 
existing cognitive structure 


• Elaborates facts, concepts, 
and rules/principles 


Retrieval from LTM to 


• 


Facilitates retrieval 


• Provides dynamic visual 


STM 


• 


Activates response organization 


reinforcement to the learner’s 




• 


Tests for understanding or misunderstanding 


response 



Levels of processing theory (Craik & lockhark, 1972) claims people process incoming information at a 
successive series or hierarchy of processing levels (stages) simultaneously based on its characteristics. Direct 
manipulation matching questions and animation-elaborated feedback were supposed to better facilitate deeper 
processing of the information from shallow to deep, and from sensory stage, to matching and pattern recognition 
stage, and then to semantic-associative stage. Dual coding theory identified three distinctive levels of processing 
within and between the verbal and visual systems: (1) representational, (2) associative, and (3) referential (Paivio, 
1986). Direct manipulation matching questions and animation-elaborated feedback were believed to better facilitate 
these three levels of processing and therefore, facilitate learning. Generative learning theory (Wittrock, 1974a, 
1974b) is concerned with how learners process information and actively construct meaning from it. Direct 
manipulation matching questions and animation-elaborated feedback should be more likely to help build two types 
of relationships identified in generative learning theory: organizational and reorganizational relationships between 
and among different parts of the external information, and integrated relationships between the external information 
and the learner’s prior knowledge. Therefore, they should be more likely to facilitate learning. 

Figure 1 provides a graphical representation of the theoretical framework for this study. 




Figure 1. Theoretical framework. Source: from Gagne, R. M. and Driscoll, M. P., Essentials of learning for 
instruction, 2/e. Published by Allyn and Bacon, Boston, MA. Copyright © 1988 by Pearson Education. Adapted by 
permission of the publisher. 
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Direct Manipulation Strategy 

Hutchins et al. (1985) suggested direct manipulation interfaces make learners feel directly engaged with the 
controlled objects, not with the programs or computers. Haag (1995) explained that “direct manipulation provides 
the learner with control over the movement and placement of graphical elements using the mouse .... This type of 
capability of the computer allows for interaction with visuals in an innovative manner” (p. 25). Grabowski (2003) 
suggested that manipulation of objects can be considered as a generative activity because it creates and extends a 
relationship between the parts of the environment. Despite the advantages of direct manipulation strategy, little 
research has been conducted to investigate its instructional effectiveness on learning. 

Haag and Grabowski (1994) investigated the effects of manipulation strategy and found learners in the 
visual summary with manipulation group performed better than those in learner manipulation or those in the 
computer manipulation groups in the problem solving test. Haag (1995) conducted an experimental study and 
examined the effects of generative and non-generative visual manipulation strategies in CBI. Four treatment groups 
included: visual summary with manipulation, learner manipulation, computer manipulation, and a control group. She 
found no main effects for manipulation strategies. The control groups, however, showed highest mean scores. The 
above studies show that there are still questions about the effectiveness of direct manipulation. Furthermore, none of 
them examined the instructional effects of direct manipulation questioning strategy compared to the traditional 
questioning strategies, such as traditional matching questions. In addition, Haag’s direct manipulation strategy 
required the learners to manipulate visual screen elements, while this study required the learners to manipulate 
verbal screen elements. 

Animation-Elaborated Feedback Strategy 

Rieber (1990) claimed that animation is a powerful way to create a wide variety of practice strategies which 
has broadened the nature of the interaction between the student and an instructional task significantly. He pointed 
out that when used in the most structured practice activities, animations were used as feedback to student answers 
providing reinforcement to correct answers. On the other hand, animation can be used in highly interactive programs 
in which animated graphics change continuously over time based on student input. 

A series of research studies (Rieber, 1996; Rieber, Tzeng, Tribble & Chu, 1996) examined the specific role 
of real-time graphic feedback and how learners select, process, and interpret real-time feedback provided by 
animation. However, these studies have been focused on using animation as continuous visual feedback in a 
simulation environment. More research needs to be conducted to investigate the instructional effects of animation 
used as a feedback to learners’ responses providing reinforcement in more structured practice activities. In this study, 
animation-elaborated feedback provided immediate informational feedback by demonstrating or elaborating a lesson 
fact, concept, rule or principle when the learner’s response was correct. 

Prior Knowledge 

Prior knowledge has been considered the most important single factor influencing learning (Ausubel, 1968). 
Jonassen and Grabowski (1993) defined prior knowledge and achievement as the knowledge, skills, or abilities 
brought by learners to the learning environment before instruction. According to Dwyer (1994), because prior 
knowledge of lower-level objectives predicts performance in higher-order objectives, it should be included in any 
study of higher-order learning. Although students were randomly assigned to different treatments, their prior 
knowledge was still tested to wash out any effects it might have on learning. 

Research Purpose and Questions 

This research study focused on the instructional effects of two specific instructional strategies: questioning 
strategies and feedback strategies. This purpose was to investigate the instructional effects of two types of 
questioning strategies (traditional matching questions, and direct manipulation matching questions) and two types of 
feedback strategies (knowledge of response (KOR) text feedback only, and KOR text plus animation-elaborated 
feedback). The traditional matching questions, direct manipulation matching questions and the animation-elaborated 
feedback were designed based on cognitive processing requirements of three levels of learning: factual, conceptual, 
and rules and principles. 

Three research questions were explored. (1) Were no questions, traditional matching questions, and direct 
manipulation matching questions equally effective in facilitating student achievement on tests measuring different 
educational objectives? (2) Were no feedback, KOR text feedback only, and KOR text plus animation-elaborated 
feedback equally effective in facilitating student achievement on tests measuring different educational objectives? (3) 
Was there an interaction between the question type and the feedback type? 
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Methods 



Participants 

Participants were 184 undergraduate volunteers enrolled in a non-major biology entry level course during the 
fall 2005 semester in a large land grant university in northeastern USA. Students in this class included freshmen 
through seniors from all kinds of majors across campus. 



Instructional Materials 

The self-paced web-based instruction used in this study was adapted from a color-coded, paper-based 
booklet developed by Dwyer and Lamberski (1977) about the human heart. The content contained approximately 
1,800 words and was divided into three sections: the parts of the heart, circulation of blood, and cycle of blood 
pressure. The paper-based instructional booklet was developed into a 21 -screen web-based instruction. Ten 
additional practice screens were designed and developed for treatments 2, 3, 4 and 5 in order to integrate questioning 
and feedback strategies into the instructional material: four practice screens were placed after the first content 
section (the Parts of the Heart), three were placed after the second content section (Circulation of Blood), and three 
were placed after the third content section (Cycle of Blood Pressure). 



Item Analysis 

Integration and positioning of the questioning and feedback strategies were determined by an item analysis 
that identified where students were having difficulties based on their performance on the criterion tests from a pilot 
study conducted with 35 students enrolled in the same Biological Science course. From the item analysis, 37 of a 
total of 60 test items were identified in the three criterion tests in which student achievement was less than or equal 
to 60% correct (5 items in the Identification test, 14 items in the Terminology test, and 18 items in the 
Comprehension test). The content referred to in the 37 test items was located on 15 instructional computer screens. 



Treatments 

Five instructional treatments were included in the present study. Treatment 1 contained 1 directions screen 
and 20 instructional screens. No questions or feedback practices were presented. Treatments 2, 3, 4, and 5 contained 
additional 10 practice screens which included questioning and feedback strategies to help students practice on the 
content referred to by the difficult test items identified through the item analysis. In each treatment, the web-based 
instruction was self-paced. 

Treatment 1: No questions and no feedback (Control group). This web-based treatment contained 21 
screens (1 directions screen and 20 instructional screens) with instructional text on the left side and the 
corresponding static graphic on the right side of each screen. No questions and feedback practices were provided. 
See example screenshot in Figure 2. 



The heart lies toward the front of the body and is in a 
slanting position between the lungs, immediately below 
the breastbone. The wide end points toward the right 
shoulder. The small end of the heart points downward to 
the front of the chest and toward the left. The lower 
portion of the heart is called the apex and is the part that 
you feel beating. 

The human heart is really two pumps combined in a 
single organ which circulate blood to all parts of the 
body. The heart is divided longitudinally into two halves 
by the septum. The two halves may be compared to a 
block of two houses, which are independent of each 
other but have a common wall, the septum, between 
them. 



Septum 




Figure 2. Example instructional screen received by the control group (Tl). 



Treatment 2: Traditional matching questions with KOR text feedback. Three types of traditional matching 
questions with KOR text feedback were embedded in the 10 additional practice screens to facilitate factual, 
conceptual, and rules and principles learning. See example screenshot in Figure 3. For facts, students were required 
to correctly identify and match the concept names with their corresponding graphical parts by clicking on the correct 
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radio button under each concept name. For concepts, students were required to correctly identify and match the 
concept attributes with their corresponding concept names by clicking on the correct radio button under each 
concept attribute. For rules and principles, students were required to correctly identify and match the “If. . .” 
statements with their corresponding “then. . .” statements by clicking on the correct radio button under each “If. . .” 
statement. Text feedback was provided in red as either “Correct” or “Wrong” at the top of the screen once the 
student clicked on a radio button. After the students correctly responded to all matching questions, the “NEXT” 
button was activated so that they could proceed to the next screen. It was expected that this type of practice strategy 
would facilitate factual, conceptual, and rules and principles learning through covert mental matching and KOR text 
feedback. 




Figure 3. Example practice screen for concepts received by T2. 



Treatment 3: Traditional matching questions with KOR text plus animation-elaborated feedback. This 
treatment was the same as treatment 2 except that when the students clicked on the correct radio button, one of the 
three types of animation-elaborated feedback was also displayed simultaneously in addition to the text feedback 
displayed in red as “Correct.” See example screenshots in Figure 4. For facts, animation highlighted the 
corresponding graphical parts of the concepts. For concepts, animation demonstrated the attributes of the 
corresponding concepts using dynamic visuals. For rules and principle, animation illustrated the cause-and-effect or 
correlational relationships between the “If. . .” and “then. . .” statements. When the wrong radio button was clicked, 
text feedback was displayed in red as “Wrong,” but no animation was played. It was expected that this type of 
practice strategy would facilitate factual, conceptual, and rules and principles learning through covert mental 
matching and KOR text plus animation-elaborated feedback. 
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Figure 4. KOR text plus animation-elaborated feedback for concepts received by T3. 



Treatment 4: Direct manipulation matching questions with KOR text feedback. Three types of direct 
manipulation matching questions with KOR text feedback were embedded in the 10 additional practice screens to 
facilitate factual, conceptual, and rules and principles learning. See example screenshot in Figure 5. For facts, 
students were required to correctly identify, and click and drag the concept names and drop them into the 
placeholders pointing to their corresponding graphical parts. For concepts, students were required to correctly 
identify, and click and drag the concept attributes and drop them into the placeholders under their corresponding 
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concept names. For rules and principles, students were required to correctly identify, and click and drag the “If. . .” 
statements and drop them into the placeholders above their corresponding “then. . .’’statements. Text feedback was 
provided in red as “Correct” at the top of the screen when the student dropped either the concept name, concept 
attribute, or “If. . .” statement into the correct placeholder. When the item was dropped anywhere else other than the 
correct placeholder, text feedback was displayed in red as “Wrong,” and the item would pop back to its original 
position providing negative animation feedback. Then the students could drag and drop it again. After the students 
correctly responded to all matching questions, the “NEXT” button was activated so that they could proceed to the 
next screen. It was expected that this type of practice strategy would facilitate factual, conceptual, and rules and 
principles learning through covert mental matching, overt direct manipulation, and KOR text feedback. 




Figure 5. Example practice screen for concepts received by T4. 



Treatment 5: Direct manipulation questioning strategy with KOR text plus animation-elaborated feedback. 
This treatment was the same as treatment 4 except that when the students dropped either the concept name, concept 
attribute, or “If. . .” statement into the correct placeholder, one of the three types of animation-elaborated feedback 
was also displayed simultaneously in addition to the text feedback displayed in red as “Correct.” The three types of 
animation-elaborated feedback were the same as those described in treatment 3. It was expected that this type of 
practice strategy would facilitate factual, conceptual, and rules and principles learning through covert mental 
matching, overt direct manipulation, and KOR text plus animation-elaborated feedback. 

Procedures 

The three-stage study included an online pretest to determine the participants’ prior knowledge, followed by 
the web-based treatments and three online criterion tests (Identification, Terminology, and Comprehension). All 
stages were completed in computer labs. The result of the human physiology pretest of the participants was used as a 
CO variate to help analyze and interpret the results. 

Measurement Instruments 

The pretest and three criterion tests were developed by Dwyer (1978). The pretest consisted of 36 multiple- 
choice questions on human physiology. The Cronbach’s alpha reliability of the pretest was .45 (n = 184). Each of 
the three criterion tests consists of 20 multiple-choice questions to measure different learning objectives. The 
Identification test measured student ability to identify the names and positions of the parts. The Terminology test 
measured student knowledge of specific facts, terminologies, and definitions. The Comprehension test measured 
student understanding of complex procedures and/or processes. There was no time limit for test completion. Each 
test item was worth 1 point for a total of 20 possible points per test. The Cronbach’s alpha reliabilities for the three 
criterion tests were all above 0.8 when data were analyzed with all test items. Further analysis with the subset of 
difficult items in each criterion test showed that the Cronbach’s alpha reliabilities were all above 0.6. 

Data analysis 

The study used an incomplete 3x3 factorial design with five treatment groups. The two factors included the 
type of questioning strategies and the type of feedback strategies. Two-way MANCOVAs were conducted with all 
test items as well as subset of difficult test items in each criterion test to test for the main effects and the interaction 
between the two factors. A significance level of .05 was set for all statistical tests. 
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Results 



Descriptive Statistics 

Table 1 presents the means and standard deviations for the three criterion tests examined by question type 
and feedback type. 



Table 1 

Means and Standard Deviations for the Three Criterion Tests by Question Type and Feedback Type (All Items) 



Test 


Question Type 


Feedback Type 


Mean 


S. D. 


n 


Identification 


All question types 


No feedback (Tl) 


13.03 


4.53 


37 






KOR text feedback (T2+T4) 


14.22 


4.61 


73 






KOR text plus animation-elaborated feedback (T3+T5) 


13.22 


4.89 


74 






All feedback types (T1+T2+T3+T4+T5) 


13.58 


4.72 


184 




No question 


No feedback (Tl) 


13.03 


4.53 


37 






Total 


13.03 


4.53 


37 




Traditional 


KOR text feedback (T2) 


14.69 


5.05 


36 




matching questions 


KOR text plus animation-elaborated feedback (T3) 


13.61 


4.47 


36 






Total 


14.15 


4.77 


72 




Direct manipulation 


KOR text feedback (T4) 


13.76 


4.16 


37 




matching questions 


KOR text plus animation-elaborated feedback (T5) 


12.84 


5.29 


38 






Total 


13.29 


4.76 


75 


Terminology 


All question types 


No feedback (Tl) 


10.22 


4.32 


37 






KOR text feedback (T2+T4) 


11.59 


4.54 


73 






KOR text plus animation-elaborated feedback (T3+T5) 


10.82 


4.68 


74 






All feedback types (T1+T2+T3+T4+T5) 


11.01 


4.56 


184 




No question 


No feedback (Tl) 


10.22 


4.32 


37 






Total 


10.22 


4.32 


37 




Traditional 


KOR text feedback (T2) 


12.56 


4.80 


36 




matching questions 


KOR text plus animation-elaborated feedback (T3) 


11.17 


4.72 


36 






Total 


11.86 


4.78 


72 




Direct manipulation 


KOR text feedback (T4) 


10.65 


4.13 


37 




matching questions 


KOR text plus animation-elaborated feedback (T5) 


10.50 


4.68 


38 






Total 


10.57 


4.38 


75 


Comprehension 


All question types 


No feedback (Tl) 


10.14 


4.17 


37 






KOR text feedback (T2+T4) 


10.52 


4.53 


73 






KOR text plus animation-elaborated feedback (T3+T5) 


9.86 


4.53 


74 






All feedback types (T1+T2+T3+T4+T5) 


10.18 


4.45 


184 




No question 


No feedback (Tl) 


10.14 


4.17 


37 






Total 


10.14 


4.17 


37 




Traditional 


KOR text feedback (T2) 


11.89 


4.49 


36 




matching questions 


KOR text plus animation-elaborated feedback (T3) 


10.39 


4.33 


36 






Total 


11.14 


4.45 


72 




Direct manipulation 


KOR text feedback (T4) 


9.19 


4.21 


37 




matching questions 


KOR text plus animation-elaborated feedback (T5) 


9.37 


4.72 


38 






Total 


9.28 


4.44 


75 



Note. Each of the criterion test scores could range from a low of 0 to a high of 20. 



Descriptively, Table 1 shows that regardless of question type, students who received KOR text feedback 
(T2+T4) had higher mean scores than students who received no feedback (Tl) or KOR text plus animation- 
elaborated feedback (T3+T5) on all three criterion tests. The mean scores of students who received KOR text plus 
animation-elaborated feedback (T3+T5) were higher than students who received no feedback (Tl) on the 
Identification test and Terminology tests, but were lower on the Comprehension test. Regardless of feedback type, 
students who received traditional matching questions (T2+T3) had higher mean scores than students who received 
no questions (Tl) or direct manipulation matching questions (T4+T5) on all three criterion tests. The mean scores of 
students who received direct manipulation matching questions (T4+T5) were higher than students who received no 
questions (Tl) on the Identification and Terminology tests, but were lower on the Comprehension test. Further 
analysis with the subset of difficult items in each criterion test showed the same trends (see Table 2). 
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Table 2 

Means and Standard Deviations for the Three Criterion Tests by Question Type and Feedback Type (Subset of 
Difficult Items) 



Test 


Question Type 


Feedback Type 


Mean 


S. D. 


n 


Identification 


All question types 


No feedback (Tl) 


2.59 


1.50 


37 


(5 Items) 




KOR text feedback (T2+T4) 


3.25 


1.49 


73 






KOR text plus animation-elaborated feedback (T3+T5) 


2.86 


1.58 


74 






All feedback types (T1+T2+T3+T4+T5) 


2.96 


1.54 


184 




No question 


No feedback (Tl) 


2.59 


1.50 


37 






Total 


2.59 


1.50 


37 




Traditional matching 


KOR text feedback (T2) 


3.36 


1.66 


36 




questions 


KOR text plus animation-elaborated feedback (T3) 


2.83 


1.58 


36 






Total 


3.10 


1.63 


72 




Direct manipulation 


KOR text feedback (T4) 


3.14 


1.32 


37 




matching questions 


KOR text plus animation-elaborated feedback (T5) 


2.89 


1.61 


38 






Total 


3.01 


1.47 


75 


Terminology 


All question types 


No feedback (Tl) 


6.35 


3.15 


37 


(14 Items) 




KOR text feedback (T2+T4) 


7.60 


3.25 


73 






KOR text plus animation-elaborated feedback (T3+T5) 


7.07 


3.47 


74 






All feedback types (T1+T2+T3+T4+T5) 


7.14 


3.33 


184 




No question 


No feedback (Tl) 


6.35 


3.15 


37 






Total 


6.35 


3.15 


37 




Traditional matching 


KOR text feedback (T2) 


8.36 


3.45 


36 




questions 


KOR text plus animation-elaborated feedback (T3) 


7.56 


3.34 


36 






Total 


7.96 


3.40 


72 




Direct manipulation 


KOR text feedback (T4) 


6.86 


2.90 


37 




matching questions 


KOR text plus animation-elaborated feedback (T5) 


6.61 


3.57 


38 






Total 


6.73 


3.24 


75 


Comprehension 


All question types 


No feedback (Tl) 


8.73 


3.64 


37 


(18 Items) 




KOR text feedback (T2+T4) 


9.23 


4.09 


73 






KOR text plus animation-elaborated feedback (T3+T5) 


8.61 


4.10 


74 






All feedback types (T1+T2+T3+T4+T5) 


8.88 


4.00 


184 




No question 


No feedback (Tl) 


8.73 


3.64 


37 






Total 


8.73 


3.64 


37 




Traditional matching 


KOR text feedback (T2) 


10.44 


4.01 


36 




questions 


KOR text plus animation-elaborated feedback (T3) 


9.08 


3.93 


36 






Total 


9.76 


4.00 


72 




Direct manipulation 


KOR text feedback (T4) 


8.05 


3.86 


37 




matching questions 


KOR text plus animation-elaborated feedback (T5) 


8.16 


4.27 


38 






Total 


8.11 


4.04 


75 



Note. The theoretical score for each criterion test was: Identification: 0 through 5; Terminology: 0 through 14; Comprehension: 0 
through 18. 



Analysis of Null Hypotheses 

Data were analyzed with all test items as well as the subset of difficult items in each criterion test. Table 3 
and table 4 showed the overall MANCOVA results using Pallai’s Trace F. 



Table 3 

Two-Way Factorial MANCOVA Overall Test Results (n=184, All Items) 



Source 


Pillai’s 

Trace 


F 


Sig 


Partial Eta 
Squared 


Observed 

Power 


Covariate Prior Knowledge 


.18 


12.57 


.000 


.18 


1.00 


Question Level 


.04 


2.17 


.094 


.04 


.54 


Feedback Level 


.01 


.66 


.579 


.01 


.19 


Question Level by Feedback Level 


.01 


.57 


.635 


.01 


.17 



Note. Each of the criterion tests contains 20 items. 
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Table 4 

Two-Way Factorial MANCOVA Overall Test Results (n=184, Subset of Difficult Items) 



Source 


Pillais 

Trace 


F 


Sig 


Partial Eta 
Squared 


Observed 

Power 


Covariate Prior Knowledge 


.17 


11.72 


.000 


.17 


1.00 


Question Level 


.05 


3.00 


.032 


.05 


.70 


Feedback Level 


.01 


.85 


.468 


.01 


.23 


Question Level by Feedback Level 


.01 


.46 


.709 


.01 


.14 



Note. The Identification test contains 5 items, the Terminology test contains 14 items, and the Comprehension test 
contains 18 items. 

The results of Pillai’s Trace showed three finding. There were no significant differences in achievement 
among students who received either no questions, traditional matching questions, or direct manipulation matching 
questions when data were analyzed using all test items in each criterion test, F(3,176) = 2.17,/? = .094 (see Table 3). 
However, further analysis with only the subset of difficult items in each criterion test found significant differences in 
achievement among students who received either no questions, traditional matching questions, or direct 
manipulation matching questions, F(3,176) = 3.00,/? = .032 (see Table 4). Follow-up univariate pairwise 
comparisons revealed that students who received traditional matching questions (T2+T3, Mean = 7.96) performed 
significantly better than students who received no questions (Tl, Mean = 6.35) on the Terminology test only,/? 

= . 020 . 

There were no significant differences in achievement among students who received either no feedback, 
KOR text feedback only, or KOR text plus animation-elaborated feedback, F(3,176) = .66,/? = .579 (see Table 3). 
Further analysis with only the subset of difficult items in each criterion test also found no significant differences, 
F(3,176) = .85,/? = .468 (see Table 4). 

There was no significant interaction between the question type and the feedback type, F (3,176) = .57,/? 

= .635 (see Table 3). Further analysis with only the subset of difficult items in each criterion test also found no 
significant interaction between the two factors, F(3,176) = .46,/? = .709 (see Table 4). 

Discussion 

Effects of Different Types of Questioning Strategies 

The results showed that traditional matching questions (T2+T3) and direct manipulation matching 
questions (T4+T5) were equally effective in facilitating student achievement in all criterion tests when data were 
analyzed with all test items as well as the subset of difficult items. The main reason might be because both 
traditional matching questions and direct manipulation matching questions may have matched the cognitive 
processing requirements. They were functionally the same, but structurally different. 

Furthermore, the results showed that students who received traditional matching questions (T2+T3) showed 
higher mean scores than those who received direct manipulation matching questions (T4+T5) on all three criterion 
tests when data were analyzed with all test items as well as the subset of difficult items. Although the differences 
were not statistically significant at the .05 alpha level, according to Huck (2004), the difference between the two 
questioning strategies was approaching marginal significance on the Comprehension test. The trends may be 
explained by the following reasons. First of all, compared to the traditional matching questions, direct manipulation 
matching questions is a “. . . novel, unconventional, demanding and possibly a distracting generative strategy for 
students” (Haag, 1995, p. 86). Secondly, the manipulation or generative reorganization of information might have 
made the recall of specific facts and concepts more difficult for some learners. This argument was corroborated by 
the results reported by Perrig (1988) and Haag (1995). Thirdly, students with higher prior knowledge or familiarity 
with the content will benefit more from a conventional instructional strategy (Haag, 1995). 

Additional results indicated that students who received no questions (Tl) performed equally well as those 
who received traditional matching questions (T2+T3) and those who received direct manipulation matching 
questions (T4+T5) on all three criterion tests when data were analyzed with all test items and on the Identification 
and Comprehension tests when data were analyzed with the subset of difficult items. This may be explained by the 
following reasons. First of all, all 184 student volunteers in this study were from a Biological Science course who 
might have been very interested in the heart content, and therefore, might have been highly motivated to learn. 
Secondly, the instructional material itself might be powerful enough that the students did not need additional 
instructional practice strategies to facilitate their learning. However, the results also indicated students who received 
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traditional matching questions (T2+T3) performed significantly better than those who received no questions (Tl) on 
the Terminology test when data were analyzed with the subset of difficult items. This suggested that students did 
benefit from traditional matching questions on learning difficult concepts. 

Effects of Different Types of Feedback Strategies 

The results indicated that no feedback, KOR text feedback only, and KOR text plus animation-elaborated 
feedback were equally effective in facilitating student achievement on the three criterion tests. The results concurred 
with those reported by Rieber and his colleagues (Rieber, 1996; Rieber et al., 1996) that graphical feedback, textual 
feedback, and graphical plus textual feedback were equally effective for students to learn explicit knowledge. 
However, the results were contradictory to what was expected based on the literature. This may be explained by the 
following reasons. First of all, compared to KOR text feedback, students were not familiar with the animation- 
elaborated feedback strategy and might not know how to attend to the relevant cues or details provided by animation. 
The support of this reasoning can be found in previous literature (i.e., Dwyer, 1978; Rieber, 1990, 2000). Secondly, 
animation-elaborated feedback might distract students’ attentions from building generative relationships instead of 
facilitating their cognitive processing. Thirdly, the characteristics of the participants and the powerful instructional 
material itself could also help explain the equal effects of the different feedback strategies. 

Conclusions 

The following conclusions have been drawn: (1) traditional matching questions and direct manipulation 
matching questions were equally effective in facilitating factual, conceptual and rules and principles learning; (2) no 
questions, traditional matching questions, and direct manipulation matching questions were equally effective in 
facilitating factual, and rules and principles learning; (3) compared to no questions, traditional matching questions 
were more effective in facilitating learning difficult concepts; (4) no feedback, KOR text feedback, and KOR text 
plus animation-elaborated feedback were equally effective in facilitating factual, conceptual, and rules and 
principles learning; (5) instructional text with corresponding static graphics is still a very powerful instructional 
strategy; (6) learners need to be guided or trained in order to possibly benefit from direct manipulation matching 
questions; and (7) additional instructional strategies are needed in order to direct learner’s attention to the most 
important information provided by animation (Rieber, 1990, 2000). 

Implications for Instructional Designers 

The following suggestions are provided for instructional designers: (1) for factual, conceptual, and rules and 
principles learning, results are no different whether traditional matching questions or direct manipulation matching 
questions are used. (2) for factual and rules and principles learning, using either no questions, traditional matching 
questions, or direct manipulation questions are equally effective on student learning; (3) for conceptual learning, 
using traditional matching questions should better facilitate learning difficult concepts versus no questions; (4) for 
factual, conceptual, and rules and principles learning, using no feedback, KOR text feedback only, and KOR text 
plus animation-elaborated feedback should be equally effective on student learning; (5) because instructional text 
with corresponding static graphics is still a very powerful instructional strategy, it may not be necessary to use 
additional instructional strategies to facilitate learning, especially when the learners have higher prior knowledge 
and are familiar with the content; (6) when direct manipulation matching questions are used, additional instructional 
strategies may be needed to guide learners; and (7) the same for using animations. Animations alone are not 
powerful enough to effect a difference in learning. 

Recommendations for Future Research 

The following recommendations are provided for future research: (1) rerun the study by providing detailed 
directions and practice for direct manipulation matching questions; (2) rerun the study by providing additional 
instructional strategies to cue learners to the most important information provided by animation-elaborated feedback; 
(3) rerun the study with a better designed animation-elaborated feedback; (4) rerun the study with students from a 
non-Biology course; and (6) investigate the effects of direct manipulation questions and animation-elaborated 
feedback on learning retention, learning efficiency, student motivation, interests and attitudes, and higher level 
learning tasks, such as problem-solving. 
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